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Accounting and Billing System," having inventors Limor Schweitzer and Eran 
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application also relates to the United States patent application having serial 
10 number XX/XXX,XXX, entitled "Network Accounting and Billing System," 
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BACKGROUND OF THE INVENTION 

A. Field of the Invention 

This invention relates to the field of computer networks. In particular, 
the invention relates to accounting and billing for services in a computer 
network. 

B. Description of the Related Art 

The low cost of Internet connectivity and a wide range of services are 
driving and more people onto the Internet, which is driving the deployment of 
TCP/IP networks. This process has led to a new market of client-server 
applications that enables the user to interact with other users and computer 
systems around the world. The use of these applications is consuming more and 
more Intranet and Internet bandwidth. 

New applications such as "voice over IP (Internet Protocol)" and 
streaming audio and video require even more bandwidth and a different quality 
of service than email, or other less real-time applications. Also, the type quality 
of service can vary according to the needs of the user. For example, typically, 
businesses do not tolerate unavailable network services as easily as consumers. 
Internet Service Providers (ISPs) therefore would like to price their available 
bandwidth according to a user's needs. For example, flat monthly pricing may 
be the best billing model for consumers, but businesses may want to be billed 
according to their used bandwidth at particular qualities of service. 

2 
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As ISPs continue to differentiate themselves by providing additional 
services, enterprise information technology managers will face similar problems 
to account for the escalating Intranet operating costs. 

Therefore, ISPs and enterprise information technology managers will 
5 want to account for session logging, bandwidth usage, directory data and 

application session information from a variety of sources. 

Due to the diversity of IP data sources (e.g., routers, hubs etc.), the need 
for effect tracking far exceeds the problems addressed by telephone companies. 
Telephone companies track information such as circuit usage so it can be 
1 0 correlated with account information. For example, businesses may use leased 

lines, consumers may have "Friends and Family" plans, cellular phones have 
different roamer fees according to the location of the user. etc. Typically, the 
phone company captures all of the data and uses batch processing to aggregate 
the information into specific user accounts. For example, all the long distance 
1 5 calls made during a billing period are typically correlated with the Friends and 

Family list for each phone account at the end of a billing period for that account. 
This requires a significant amount of computing power. However, this type of 
problem is significantly simpler than attempting to track and bill for every 
transaction in an IP network. Therefore, what is desired is a system that allows 
20 for accounting and billing of transactions on IP based networks. 

The problem is even more difficult in IP network traffic because the 
information sources can exist and many different levels of the OSI network 
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model, throughout heterogeneous networks. Potential sources of information 
include packet use from routers, firewall authentication logging, email data, ISP 
session logging, and application layer use information. Therefore, what is 
desired is a system and method that track IP network usage information across 
5 multiple layers of the OSI network model. 
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SUMMARY OF THE INVENTION 

A network accounting and billing system and method are described. In 
some embodiments, the system can access any network related information 
sources such as traffic statistics provided by routers and switching hubs as well 

5 as application server access logs. The information can be accumulated in a 

central database for creating auditing, accounting and billing reports. 
Alternatively, the information can be sent directly to other systems such as 
rating engines used in customer care and billing systems. 

In one embodiment, network traffic information is captured at network 

1 0 information sources (examples of information sources include network devices). 

These sources provide detailed information about the network communications 
transactions at a network device. Importantly, different types of sources can 
provide different types of information. Gatherer devices gather the detailed 
information from the various information source devices and convert the 

15 information into standardized information. The gatherer devices can con-elate 

the gathered information with account information for network transaction 
accounting. Manager devices manage the gatherer devices and store the 
gathered standardized information. The manager devices eliminate duplicate 
network information that may exist in the standardized information. The 

20 manager devices also consolidate the information. Importantly, the information 

stored by the manager devices represents the consolidated, account correlated. 
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network transaction information used for billing. In addition to account 
information, transaction information can be correlated to other information such 
as geography information (e.g., the location of an accessed server) and/or 
transaction routing information (as may be used in peering agreements between 
5 Internet Service Providers). The system thereby provides a distributed network 

accounting and billing system. 

In some embodiments, the gatherer devices can access sources through 
proxy gateways, firewalls, and/or address translation barriers. 

In some embodiments, the gatherer devices can correlate the information 
10 about a specific transaction with a particular account by accessing the 

transaction's source and/or destination information. The source and/or 
destination information is then correlated with account information from an 
account information database. 



6 
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BRIEF DESCRIPTION OF THE FIGURES 

The figures illustrate the invention by way of example. The invention is 
not meant to be limited to only those embodiments of shown in the Figures. The 
same reference in different figures indicates the same element is being used in 

5 those figures. 

Figure 1 illustrates a system including one embodiment of the invention. 
Figure 2 illustrates an example of the data distillation used in the system 
of Figure 1. 

Figure 3 illustrates data enhancements used in the data distillation. 

0 Figure 4 A illustrates example field enhancements that can be included in 
the data enhancements. 

Figure 4B illustrates the creation of an enhanced record. 
Figure 5 illustrates an example record merge. 

Figure 6 illustrates an example of an alternative embodiment of the 

1 5 . system. 
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DETAILED DESCRIPTION 



A. System Overview 

One embodiment of the system includes a multi-source, multi-layer 
network usage metering and mediation solution that gives Network Service 
Providers (NSPsX including Internet Service Providers (ISPs) and enterprise 
network(Intranet) operators, the information needed to set the right-price for IP 
(Internet Protocol) services. With the system, the providers can generate 
accurate usage-based billing and implement usage-based charge-back models. 
The system derives IP session and transaction information, collected in real 
time, from a multitude of network elements. The system gathers, correlates, and 
transforms data from routers, switches, firewalls, authentication servers, LDAP, 
Web hosts, DNS, and other devices to create comprehensive usage and billing 
records. 

The system transforms raw transaction data from network devices into 
useful billing records though policy-based filtering, aggregation, and merging. 
The result is a set of detail records (DRs). In some embodiments, the detail 
records are XaCCT Detail Records (XDRs™) available from XaCCT 
Technologies. DRs are somewhat similar in concept to the telephony industry's 
Call Detail Records (CDRs). Thus, DRs can be easily integrated with existing 
Customer Care and Billing (CCB) systems. 

8 
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In addition to billing data. DRs enable NSPs to deploy new services 
based on documented usage trends, plan network resource provisioning, and 
audit service usage. The system provides a clear picture of user-level network 
service use by tracking a variety of metrics such as actual session Quality of 
5 Service (QoS),traffic routes, and end-user application transactions. 

The system is based on a modular, distributed, highly scalable 
architecture capable of running on multiple platforms. Data collection and 
management is designed for efficiency to minimize impact on the network and 
system resources. 

1 0 The system minimizes network impact by collecting and processing data 

close to its source. Modular architecture provides maximum configuration 
flexibility, and compatibility with multiple network information sources. 

The system, or other embodiments, may have one or more of the 
following features. 

1 5 Data collection can be from a wide range of network devices and 

services, spanning all layers of the network - from the physical layer to the 
application layer. 

Real-time, policy-based filtering, aggregation, enhancement and 
merging creates accurate, detailed and comprehensive session detail records 

20 (DRs). 

Real time correlation of data from various sources allows billing record 
enhancement. 
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Leverages existing investment through integration with any customer 
care & billing solution, reducing costs, minimizing risks and shortened time-to- 
market. 

Non-intrusive operation eliminates any disruption of network elements 
5 or services. 

Web-based user interface allows off-the-shelf browsers to access the 
system, on-demand, locally or remotely. 

Carrier-class scalability allows expansion to fit an NSPs needs without 
costly reconfiguration. 
1 0 Distributed filtering and aggregation eliminates system capacity 

bottlenecks. 

Efficient, centralized system administration allows on-the-fly system 
reconfigurations and field upgrades. 

Customized reporting with built-in report generation or an NSPs choice 
15 of off-the-shelf graphical reporting packages. 

Comprehensive network security features allow secure communication 
between system components and multiple levels of restricted access. 

B. System Details 

The following describes the system 100 of Figure 1. The system 100 
20 allows NSPs to account for and bill for IP network communications. The 

following paragraphs first list the elements of Figure 1 . then describes those 
elements and then describes how the elements work together. Importantly, the 

10 
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distributed data gathering, filtering and enhancements performed in the system 
100 enables load distribution. Granular data can reside in the peripheries of the 
system 100, close to the information sources. This helps avoids reduce 
congestion in network bottlenecks but still allows the data to be accessible from 
5 a central location. In previous systems, all the network information flows to one 

location, making it very difficult to keep up with the massive record flows from 
the network devices and requiring huge databases. 

The following lists the elements of Figure 1 . Figure 1 includes a number 
of information source modules (ISMs) including an ISM 1 10, an ISM 120, an 
10 ISM 130. an ISM 136, an ISM 140, and an ISM .150. The system also includes a 

number of network devices, such as a proxy server 101 , a DNS 102, a firewall 
103, an LDAP 106, a CISCO NetFlow 104, and a RADIUS 105 . The system 
also includes a number of gatherers, such as a gatherer 161, a gatherer 162, a 
gatherer 163, a gatherer 164, and a gatherer 165. The system of Figure 1 also 
1 5 includes a central event manager (CEM) 1 70 and a central database (repository) 

1 75. The system also includes a user interface server 1 85 and a number 
terminals or clients 1 80. 

This paragraph describes how the elements of Figure 1 are coupled. The 
various network devices represent devices coupled to an IP network such as the 
20 Internet. The network devices perform various functions, such as the proxy 

server 1 0 1 providing proxy service for a number of clients. Each network device 
is coupled to a corresponding ISM. For example, the proxy server 101 is 
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coupled to the ISM 1 10. The DNS 102 is coupled to the ISM 120. The firewall 
103 is coupled to the ISM 130. The ISM 136 is coupled to the LDAP 106. The 
ISM 140 is coupled to the CISCO NetFlow 104. The ISM 150 is coupled to the 
RADIUS 105. Each gatherer is associated with at least one ISM. Thus, the 
5 gatherer 1 6 1 is associated with the ISM 1 1 0 and is therefore coupled to that 

ISM. The gatherer 162 is coupled to the ISM 120. The gatherer 163 is coupled 
to the ISM 130 and the ISM 136. The gatherer 164 is coupled to the ISM 140. 
The gatherer 165 is coupled to the ISM 150. The various gatherers are coupled 
to the CEM 1 70. The user interface server is coupled to the terminals 180 and 
10 the CEM 170. 

The following paragraphs describe each of the various elements of 

Figure 1. 

Network Devices 

The network devices represent any devices that could be included in a 
1 5 network. (Throughout the description, a network device, unless specifically 

noted otherwise, also refers to an application server.) A network device 
. represents a subset of information sources that can be used by the system 100. 
That is, the network devices are merely representative of the types of sources of 
information that could be accessed. Other devices such as on-line transaction 
20 processing databases can be accessed in other embodiments of the invention. 

Typically, the network devices keep logging and statistical information about 
their activity. A network information source can be the log file of a mail server. 
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the . logging facility of a firewall, a traffics statistics table available on a router 
and accessible through SNMP, a database entry accessible through the Internet, 
an authentication servers query interface, etc. The network devices represent 
^ the information sources accessed by the ISMs. 

5 Each type of network device can be accessing using a different method 

or protocols. Some generate logs while others are accessible via SNMP, others 
have proprietary APIs or use other protocols. 

ISMs 

The ISMs act as an interface between the gatherers and the network 
1 0 devices enabling the gatherers to collect data from the network devices. Thus, 

the ISMs represent modular, abstract interfaces that are designed to be platform- 
neutral. The information source modules act as interfaces or "translators", 
sending IP usage data, in real time, from the network devices to the gatherers. 
Each ISM is designed for a specific type of network data source. (In other 
1 5 embodiments, some ISM are generic in that they can extract information from 

multiple network devices). ISMs can be packaged separately, allowing NSPs to 
customize ISM configurations to meet the specific requirements of their 
network. For example, in the system of Figure 1 , if the NSP did not have Cisco 
NetFlow devices, then the ISM 140 would not have to be included. 
20 The ISMs can communicate with its corresponding network device using 

protocols and formats such as UDP/IP, TCP/IP. SNMP, telnet, file access. 
ODBC, native API. and others. 

13 
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In some embodiments,. the reliability of system 100 is enhanced through 
on-the-fly dynamic reconfiguration, allowing the NSP to add or remove 
modules without disrupting ongoing operations. In these embodiments, the 
CEM I 70 can automatically update the ISMs. 
5 The following ISMs are available in some embodiments of the 

invention. 

• Categorizer - Classifies a session to a category according to user- 
defined Boolean expression. 

• DNS (e.g. ISM 1 20) - Resolves host names and IP addresses. 

1 0 • Generic Proxy Server (e.g., ISM 1 1 0) - Collects data from access 

logs in a common log format. 

• Port / Protocol Resolution - Converts protocol/port information to 
account names and vice versa. 

• Checkpoint Fire Wall- 1 - Collects data from Fire Wall- 1 accounting 
15 log and security log. 

• Cisco IOS IP Accounting - Collects accounting data from a Cisco 
router using IOS IP accounting. 

• Cisco NetFlow Switching - Collects session data from a Cisco router 
via NetFlow switching. 

20 • NETRANET - Collects information from a standard network device. 

• Netscape Proxy Server - Collects data from a Netscape Proxy Server. 
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• Microsoft Proxy Server - Collects data from a Microsoft Proxy 

Server. 

ISMs can be synchronous, asynchronous or pipe. 
The data from an asynchronous ISM is dynamic so that the 
5 asynchronous ISM reacts to the information and relays it to the associated 

gatherer without prompting from other information sources in the system 100. 
If the firewall 103 were a Checkpoint Fire Wall- 1 , then the ISM 130 would be 
an example of an asynchronous ISM. When a network session is initiated, the 
details are recorded by the F ire Wall- 1 103. The corresponding ISM 130 receives 
10 the details and passes them on automatically to the gatherer 163. 

Synchronous ISMs provide its information only when accessed by a 
gatherer. The ISM 120 is an example of a synchronous ISM. The DNS server 
102 maintains information matching the IP addresses of host computers to their 
domain addresses. The ISM 120 accesses the DNS server 102 only when the 
1 5 ISM 1 20 receives a request from the gather 162. When the DNS server 102 

returns a reply, the ISM 120 relays the reply information to the gatherer 162. 

Pipe ISMs operate on record flows (batches of records received from 
information sources). Pipe ISMs process one or more enhancement flows the 
records as the flows arrive. The pipe ISM may initiate new record flows or may 
20 do other things such as generate alerts or provision network elements to provide 

or stop services. The pipe is implemented as an ISM to keep the internal 
coherency and logic of the architecture. (Record flows can terminate in a 
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database or in a pipe ISM. The, pipe ISM can perform filtering and aggregation, 
send alarms, or act as a mediation system to provision network elements when 
some event occurs or some accumulated value is surpassed. Specifically, pipe 
ISMs can act to enable pre-payment systems to disable certain services such as a 

5 voice IP call, when the time limit is surpassed or amount of data is reached.) 

The gatherers can include caches and buffers for storing information 
from the ISMs. The buffers allow the gatherers to compensate for situations 
where there is a loss of connection with the rest of the system 100. The cache 
sizes can be remotely configured. The cache minimizes the number of accesses 

10 to the Information Source. 

ISM queries can be cached and parallelized. Caching of synchronous 
ISM queries provides for fast responses. Parallelizing queries allows for 
multiple queries to be processed at the same time. 

Gatherers 

1 5 The gatherers gather the information from the ISMs. In some 

embodiments, the gatherers are multi-threaded, lightweight, smart agents that 
run on non-dedicated hosts, as a normal user application on Windows NT or 
Unix, as a background process, or daemon. What is important though is that the 
gatherers can be any hardware and/or software that perform the functions of a 

20 gatherer. 
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The gatherers can be installed on the same network segment as the 
network device such as router and switch or on the application server itself. This 
placement of a gatherer minimizes the data traffic impact on the network. 
The gatherers collect network session data from one or more ISMs. 
5 Session data can be sent to another gatherer for enhancement or to the CEM 170 

for merging and storing in the central database 170. The gatherers can be 
deployed on an as needed basis for optimal scalability and flexibility. 

The gatherers perform flexible, policy-based data aggregation. 
Importantly, the various types of ISMs provide different data and in different 
1 0 formats. The gatherers normalize the data by extracting the fields needed by the 

CEM 1 70 and filling in any fields that may be missing. Thus, the gatherers act 
as a distributed filtering and aggregation system. The distributed data filtering 
and aggregation eliminates capacity bottlenecks improving the scalability and 
efficiency of the system 1 00 by reducing the volume of data sent on the network 
1 5 to the CEM 1 70. 

Aggregation can be done by accumulating groups of data record flows, 
generating a single data record for each group. That single record then includes 
the aggregated information. This reduces the flow of the data records. 

Filtering means discarding any record that belongs to a group of 
20 unneeded data records. Data records are unneeded if they are known to be 

collected elsewhere. A policy framework enables the NSP to configure what to 
collect where. 
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Filtering and/or aggregation can be done at any point along a data 
enhancement (described below) so that aggregation schemes can be based on 
enhanced data records as they are accumulated. The filtering and/or aggregation 
points are treated by the system 100 as pipe ISMs which are flow termination 
5 and flow starting points (ie: like an asynchronous ISM on the starting end and 

like a database on the terminating end). Data enhancement paths and filtering 
and/or aggregation schemes can be based on accumulated parameters such as 
user identification information and a user's contract type. 

As noted above, the PISM can be used in the context of filtering and/or 

1 0 aggregation. One or more record flows can terminate at the PISM and can be 

converted into one or more new record flows. Record flows are grouped based 
on matching rules that apply to some of the fields in the record flows, while 
others are accumulated or undergo some other operation such as " maximum" or 
" average" . Once the groups of accumulated records have reached some 

1 5 threshold, new accumulated records are output. This can be used for example in 

order to achieve a business-hybrid filtering and aggregation data reduction by 
imposing the business rules or the usage-based products that are offered to the 
customer, onto the record flows as they are collected in real-time. This is done 
instead of previous system where the information is stored in a database and 

20 then database operations are performed in order to create bills or reports. The 

filtering and aggregation reduces the amount of data that is stored in the central 
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database 1 75 while not jeopardizing the granularity of data that is necessary in 
order to create creative usage-based products. 

Typically, data collected from a single source does not contain all. the 
information needed for billing and accounting, such as user name and 

5 organization. In such cases, the data is enhanced. By combining IP session data 

from multiple sources, such as authentication servers, DHCP and Domain Name 
servers, the gatherers create meaningful session records tailored to the NSP's 
specific requirements. In the example of Figure 1, the gatherer 161 can provide 
information to the gatherer 1 62 so that the source IP address for an Internet 

10 session from the proxy server 101 can be combined with the domain address 

from the DNS server 102. 

The enhancement procedure can be triggered by an asynchronous ISM. 
The information from the asynchronous ISM is associated with field 
enhancements in the central database 175. A field enhancement defines how a 

1 5 field in the central database is filled from the source data obtained from the 

asynchronous ISM. Through the field enhancements, the missing parameters are 
added to a record using the data collected from one or more synchronous ISMs. 
Enhancements are described in detail below. 

The gatherers can include caches and buffers for storing information 

20 from the ISMs. The buffers allow the gatherers to compensate for situations 

where there is a loss of connection with the rest of the system 100. The caches 
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can reduce the number of accesses to an information source. The buffer and/or 

cache sizes can be remotely configured. 
Central Event Manager fCEM) 

The Central Event Manager (CEM) 170 acts as the central nervous 
5 system of the system 100, providing centralized, efficient management and 

controls of the gatherers and the ISMs. 

The CEM 1 70 can perform one or more of the following tasks: 
• Coordinates, controls, and manages the data collection process. The 
CEM 170 coordinates the operation of the gatherers and manages the 
1 0 flow of data through the system 1 00 through the collection scheme 

defined in the system configuration. The latter includes the 
configuration of the gatherers, the ISMs, the network devices, the 
fields in the central database 175 (described below), and the 
enhancement procedures. Based on the collection scheme the CEM 
15 1 70 determines the system lOO's computation flow (the set of 

operations the system 1 00 must perform to obtain the desired 
information). The CEM 170 then controls all the gatherers, 
' instructing them to perform, in a particular sequence, the operations 
defined in the computation flow. The CEM 170 receives the records 
20 collected by the gatherers and stores them in the central database 

1 75. NSPs can configure the CEM 170 to merge duplicate records 
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before storing them in the central database 175. Record merging is 
described below. 

Performs clean-up and aging procedures in the database 175. The 
system 100 collects and stores large amounts of session information 
every day. The CEM 170 removes old data to free space for new data 
periodically. The NSP defines the expiration period for the removal 
of old records. The CEM 1 70 is responsible for coordinating the 
removal of records from the central database 175. The CEM 170 
places a time stamp on every record when the record enters the 
central database 175 and deletes the record after the time period the 
NSP has defined elapses. 

Provides centralized system-wide upgrade, licensing, and data 
security . The NSP can perform version upgrades of the system 100 
at the CEM 1 70. The gatherers can be automatically upgraded once a 
new version is installed on the host computer of the CEM 170. ISMs 
are also installed via the CEM 1 70 and exported to the gatherers. The 
CEM 1 70 maintains a list of licenses installed in the system and 
verifies periodically if the system is properly licensed. This feature 
lets the NSP centrally install and uninstall licenses. It also prevents 
unlicensed use of the system 100 and any of its components. 
Monitors the state of the gatherers and ISMs. The gatherers 
periodically communicate with the CEM 170. The CEM 170 
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cominuously monitors the state of each gatherer and network devices 
in the system 100. The CEM 170 can be fault-tolerant, that is, it can 
recover from any system crash. It coordinates the recovery of the 
system 100 to its previous state. 
5 In some embodiments, a key directory server is associated with the CEM 

170. To transfer less data between the elements of the system 100, it is desirable 
that each piece of data to carry little descriptive data. For example, if IP address 
data is transferred between a gatherer and the CEM 170, a description of the IP 
address data is typically included. In some embodiments, data name/key, type, 
10 and length descriptions are included with the actual IP address data. In other 

embodiments, there the key directory server reduces the amount of descriptive 
information being sent. Every key in the directory server has a type and a 
length. Fields can be identified as variable length. Therefore, data type 
information need not be transmitted between elements in the system 100 if the 
1 5 elements use a common reference key stored in the directory server. Returning 

to the IP address data, by using the key directory server, elements need only 
send two bytes for the key id and four bytes for the actual address. Most of the 
data being sent in the system is relatively short in length. Therefore, the 
directory server helps reduce the amount of information being sent between the 
20 elements in the system 100. 

Keys can be added to the directory server. The directory server can 
therefore support expansion of the kinds of fields being sent by allowing system 
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elements to update their locally stored key ids. For example, after a recipient 
receives a record with an i; unknown" key, it contacts the directory server to get 
the key definition. 

Central Database 

5 The central database 175 is the optional central repository of the 

information collected by the system 1 00. The central database 1 75 is but one 
example of a sink for the data generated in the system 100. Other embodiments 
include other configurations. The central database 175 stores and maintains the 
data collected by the gatherers, as well as the information on the configuration 

10 of the system 100. Thus, in configuring the system 100 : theNSP defines what 

data will be stored in each field in the central database 175 and how that data is 
collected from the ISMs. 

The information on network sessions is stored in the database in the 
form of a table. Each field in the table represents a network session parameter. 

15 Each record describes a network session. The system 100 has a set of pre- 

defined fields that are configured by the CEM 170 on installation. The NSP can 
modify the central database 1 75 structure by adding, deleting, or modifying 
fields. The NSP access the data in the central database 175 by running queries 
and reports. The old data is removed from the central database 175 to free space 

20 for new data periodically. You can specify the time interval for which records 

are stored in the central database 1 75. The structure of the central database 1 75 
with some of the predefined fields is illustrated in the following figure. 
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As each IP session may generate multiple transaction records, during the 
merge process the CEM 1 70 identifies and discards duplications, enhancing the 
efficiency of the data repository. Generally, data records are passed through the 
merger program, in the CEM 170, into the central database 175. However, the 
5 data records are also cached so that if matching records appear at some point, 

the already stored records can be replaced or enhanced with the new records. 
The database tables that contain the record flows can be indexed, enhancing the 
efficiency of the data repository. A merge is achieved by matching some of the 
fields in a data record and then merging the matching records from at least two 

1 0 record flows, transforming them into one record before updating the central 

database 1 75. In some embodiments, adaptive tolerance is used to match 
records. Adaptive tolerance allows for a variation in the values of fields that are 
compared (e.g., the time field value may be allowed to differ by some amount, 
but still be considered a match.). The adaptive aspect of the matching can 

15 include learning the appropriate period to allow for the tolerance. The reason 

that the records that do not match any previous records are sent through into the 
central database 1 75, in addition to being cached for later matching, is to avoid 
loss of data in case of system failure. 

The following table illustrates an example of the types of records stored 

20 in the central database 175 by the CEM 170. 
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The system 100 supports a non-proprietary database format enabling the 
central database 1 75 to run on any of a number of commercially available 
databases (e.g., MS-SQL Server , Oracle Server, DB2 ? etc.). 

5 User Interface Server and Clients 

The User Interface Server (UIS) 185 allows multiple clients (e.g. 
terminals 1 80) to access the system 1 00 through, the Microsoft Internet 
Explorer with Java™ Plug-in or Netscape Navigator with Java™ Plug-in. Other 
embodiments can use other applications to access the system 100. The main 

1 0 function of the UIS 1 85 is to provide remote and local platform independent 

control for the system 100. The UIS 185 can provide these functions through 
windows that correspond to the various components of the system 100. Access 
to the system 100 can be password protected, allowing only authorized users to 
log in to the system and protecting sensitive information. 

1 5 The NSP can perform one or more of the following main tasks through 

the UIS 185: 

• Configure the system 1 00. 

• Create and run queries and reports on network activity and resource 
consumption. 
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• Register and license the system 100. 

C. Data Distillation 

Figure 2 illustrates the data distillation process performed by the system 
of Figure 1 . The data distillation aggregates and correlate information from 
5 many different network devices to compile data useful in billing and network 

accounting. 

First, the ISMs 210 gather data from their corresponding network 
device. Note that for some ISMs (e.g. pipe ISMs), real-time, policy-based 
filtering and aggregation 215 can also be done. This data is then fed to the 

1 0 gatherers 220. The gatherers 220 perform data enhancement to complete the 

data from the ISMs 210. The results are provided to the CEM 170. The CEM 
1 70 performs data merges 270 to remove redundant data. The merged data is 
then optionally stored in the central database 175 as a billing record 275 or is 
sent directly to an external system. The billing record information can be 

15 accessed from external applications, through the application interface 290, via a 

data record 280. Filtering and/aggregation and/or data enhancements can be 
done at any stage in the system 1 00. 

D. Data Enhancement 

As mentioned above, the gatherers 220 provide data enhancement 
20 features to complete information received from the ISMs 210. The following 
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describes some example data enhancement techniques used in some 
embodiments of the invention. 

Figure 3 illustrates an example of data enhancement. Data enhancement 
comprises a number of field enhancements. A field enhancement specifies how 
5 the data obtained from the trigger of the enhancement procedure is processed 

before it is placed in a single field in the central database 175. The.data can be 
placed in the field directly, or new information may be added to the record by 
applying a Synchronous ISM function. (In the example below, the function is 
"resolve the IP address to a host FQDN"). Field enhancements may involve one 
10 or multiple steps. There is no limit to the number of steps in a Field 

Enhancement. The data record starts with fields obtained from an asynchronous 
ISM 300. The fields in the DR 300 are then enhanced using the field 
enhancements. The enhanced fields result in the DR 320. 

A visual representation of an enhancement can be presented to the NSP. 
1 5 The enhancement may include an itinerary of ISMs starting off with an AISM 

passing through PISMs. and terminating in the CEM 170. Using this view of the 
system 100, the NSP need not be shown the actual flow of data since the flow- 
may be optimized later in order to achieve better performance. This is more of a 
graphical logical view of how the enhancement is achieved in steps. (PISMs can 
20 terminate more than one flow and initiate more than one flow.) 

A visual representation of a field enhancement shows the per-field flow 
of data correlation. This process ends in the CEM 1 70 or in a PISM. The NSP 
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supplies information telling the system 100 how to reach each of the terminating 
fields (in the CEM 170 or the PISM) starting off from the initiating fields 
(PISM or AISM). Each step of enhancement defines cross correlation with some 
SISM function. 

5 Figure 4 A illustrates various field enhancements (4 1 0 through 440). A 

field enhancement includes applying zero or more functions to a field before 
storing the field in a specified field in the central database 175. 

One-step Field Enhancement 410. The initial source data from the 
asynchronous ISM is placed directly in a field in the central database 175. 

10 Example: the field enhancement for the Source IP field. 

Two-step Field Enhancement 420. The initial source data from the 
asynchronous ISM is used to obtain new additional data from a synchronous 
network device and the new data is placed in a field in the central database 175. 
Example: the field enhancement for the Source Host field. 

1 5 Three-step Enhancement 430. The initial source data from the 

asynchronous ISM is used to obtain additional data from a synchronous ISM. 
The result is used to obtain more data from another ISM and the result is placed 
in a field in the central database 175. 

The following illustrates an example data enhancement. Suppose the 

20 data obtained from a proxy server 101 contains the source IP address of a given 

session, such as 199.203.132.2, but not the complete domain address of the host 
computer (its Fully Qualified Domain Name), such as www.xacct.com. The 
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name of the host can be obtained by another network device - the Domain Name 
System (DNS 102) server. The DNS server 102 contains information that 
matches IP addresses of host computers to their Fully Qualified Domain Names 
(FQDNs). Through an enhancement procedure the information collected from 
5 the proxy server 101 can be supplemented by the information from the DNS 

1 02. Therefore, the name of the host is added to the data (the data record) 
collected from the proxy server 101 . The process of adding new data to the data 
record from different network devices can be repeated several times until all 
required data is collected and the data record is placed in the central database 
10 :. 175. 

Figure 4B illustrates another example data enhancement where an 
enhanced record 490 is created from an initial netflow record 492. Fields in the 
enhanced record 490 are enhanced from the radius record 494, the QoS policy 
server record 496, the NMS DB record 498, and the LDAP record 499. 

15 Defining Enhancem ent Procedures 

The following describes the process for defining enhancement 

procedures in some embodiments of the system. Typically defining an 
enhancement procedures for the system 100 includes (1) defining enhancement 
procedures for each asynchronous ISM and (2) configuring field enhancements 
20 for all fields in the central database 175 for which the NSP wants to collect data 

originating from an asynchronous ISM that triggers the corresponding 
enhancement procedure. 

29 



SUBSTITUTE SHEET (RULE 26) 



PCT/US98/24963 

An enhancement procedure can be defined as follows: 

1 . Access the CEM 1 70 using the UIS 1 80. 

2. Select the enhancement procedures list using the UIS 1 80. 

3. Define the name of the new enhancement procedure. 

4. Select a trigger for the new enhancement procedure. The trigger can 
correspond to any asynchronous ISM in the system 100. 
Alternatively, the trigger can correspond to any asynchronous ISM 
in the system 1 00 that has not already been assigned to an 
enhancement procedure. 

5. Optionally, a description for the enhancement procedure can be 
provided. 

6. The new enhancement procedure can then be automatically 
populated with the existing fields in the central database 175. 
Optionally, the NSP can define the fields (which could then be 
propagated to the central database 175). Alternatively, based upon 
the type of asynchronous ISM. a preset set of fields could be 
proposed to the NSP for editing. What is important is that the NSP 
can define field procedures to enhance the data being put into the 
data records of the central database 1 75. 

7. The NSP can then define the field enhancements for every field in 
the new enhancement procedure for which the NSP wants to collect 
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data from the ISM that is the trigger of the new enhancement 
procedure. 

Defining Field Enhancements 

Defining a field enhancement involves specifying the set of rules used to 
5 fill a database field from the information obtained from the trigger of the 

enhancement procedure. The NSP defines field enhancements for each field in 
which NSP wants to collect data from the trigger. If no field enhancements are 
defined, no data from the trigger will be collected in the fields. For example, 
suppose the firewall asynchronous ISM 130 that triggers an enhancement 
10 procedure. Suppose the central database 175 has the following fields: source IP, 

source host, destination IP, destination host, user name, total bytes, service, 
date/time, and URL. If the NSP wants to collect session data for each field 
except the URL from the firewall ISM 130, which triggers the enhancement 
procedure, the NSP defines a field enhancement for each field with the 
1 5 exception of the URL. 

in some embodiments, the field enhancements are part of the 
enhancement procedure and the NSP can only define and modify them when the 
enhancement procedure is not enabled. 

The field enhancements can be defined in a field enhancement 
20 configuration dialog box. The field enhancement configuration dialog box can 

have two panes. The first displays the name of the enhancement procedure, the 
name of its trigger, and the name and data type of the field for which the NSP is 
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defining the field enhancement.. The second is dynamic and interactive. Its 
content changes depending on the NSP's input. When first displayed, it has two 
toggle buttons, End and Continue, and a list next to them. The content of the list 
depends on the button depressed. 
5 When End is depressed, the list contains all output fields whose data 

type matches the data type of the field for which the NSP is defining the field 
enhancement. For example, if the field's data type is IP Address, the list 
contains all fields that are of the same type, such as source IP and destination IP 
that the AISM supplies. The fields in the list can come from two sources: (1) the 
1 0 source data which the gatherer receives from the trigger and (2) the result 

obtained by applying a synchronous ISM function as a preceding step in the 
field enhancement. The following notation is used for the fields: 

OutputFieldName for the output of a field origination from the trigger 
SISName.FimctionName(InputArgument). Output Field for the output of a 
1 5 field that is the result of applying a function 

SISNamc.OutputField for the output of a field that is the. result of 
applying a function as the final step of a field enhancement 
The following examples are presented. 

Source IP is the field provided by the trigger of the enhancement 
20 procedure that contains the IP address of the source host. 

DNS.. .Host Name and DNS.Name(Source IP). Host name are the names 
of a field originating from the resolved function Name of a network device 
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called DNS that resolves the IP address to a domain address. The input 
argument of the function is the field provided by the trigger of the enhancement 
procedure, called source IP. It contains the IP address of the source host. The 
function returns the output field called Host Name that contains the domain 

5 address of the source host. The notation DNS. ..Host Name is used when the 

field is the result of applying the function as the final step of a field 
enhancement. The notation is DNS.Name(Source IP).Host Name is used when 
the field is used as the input to another function. 

In the user interface, if End is unavailable, none of the output fields 

1 0 matches the data type of the field. 

When Continue is depressed, the list contains all applicable v fiinctions of 
the available synchronous network device configured in the system 1001. If the 
preceding output does not match the input to a function, it cannot be applied and 
does not appear on the list. . 

1 5 The following notation is used for the functions: 

SISName.FunctionName(InpiitFieldName:InputFieldDataType) 

(Output FieldName: Output FieldDataType) 

When the function has multiple input and/or output arguments, the 
notation reflects this. The arguments are separated by commas. 
20 The following example shows a field enhancement. 

DNS. Address(Host Name:String) -> (IP Address:IP Address) 
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Where DNS is the name of the synchronous ISM (or network device) as 
it appears in the system configuration. 

Address is the name of the function. 

(Host Name:String) is the input to the function - host FQDN of data type 

5 String 

(IP Address:iP Address) is the output - IP address of data type IP 
Address 

The NSP can define the field enhancement by choosing items from the 
list. The list contains the option <none> when the End button is depressed. 

10 Choosing this option has the same effect as not defining a field enhancement: no 

data from the trigger will be stored in the field in the central database 175. 

At your request, here is one paragraph describing the algorithm(s) 
transforming the user representation of an Enhancement Procedure to an actual 
Computation Flow performed by the system. The source code accompanying 

15 this actually implements most of this. Of course, that code relies on the rest of 

the software (lots of other code), that may be required to completely understand 
it. 

Field Enhancement Computation 
In some embodiments, an enhancement procedure (EP) defines a flow of 
20 data records in the system 1 00 ? originating from an AISM that serves as a 

trigger, and terminating at the a database table, or a PISM (Pipe ISM - has the 
behavior of the database of being assigned input keys in much the same way. 
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but being installable/replacable logic in the system after it has been deployed), 
known as the Target of the EP. An EP is represented to the user as a collection 
of Field Enhancements (FEs). Each FE identifies what is performed in order to 
obtain a value (or not, if it is not defined), for that Field (either database field, or 
5 input key to PI SM), using values originating from the Trigger AISM, possibly, 

utilizing functions of SISMs available in the system. The NSP defines every FE 
independently of other FEs, and completely defines what value is placed in the 
defined field. 

In contrast to the above description of the presentation to the user of an 
1 0 EP. the system 100 (e.g., the CEM 170) computes what is called a Computation 

Flow, which is the actual operational instructions and parameters that are 
instructed to the gatherer's to achieve the desired NSP result (as defined in the 
EP). The Computation Flow is the detailed instructions required to achieve a 
"snowball" affect, of having data originating from a Trigger (AISM) in one 
1 5 gatherer . undergo specific enhancements by adding results from invocations of 

functions of SISMs with specific input parameters on the same gatherer or other 
gatherers (as required), and removing unused fields along this same path, 
eventually storing the user-configured results in the user-configured fields in an 
optimized fashion. The optimizations applied are: 
20 • Transfer minimal data between any elements in the system; 
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• Do not compute the same function on the same input more than 
once. In other embodiments, this optimization may be changed to 
allow for sharing of intermediary values of field enhancements; 

• Perform local enhancements before going on to a remote gatherer; 
5 • Apply a Si cost" function to transferring along communication links 

and traverse the minimal cost in a computation flow required to 
achieve correct results. 
This set of optimizations can be performed, in some embodiments, using 
the code in Appendix A. That is, the transformation (from EP to Computation 
1 0 Flow, or simply Flow) is achieved by coupling together eleven algorithms and 

data transformations one after the other. These are described in the code module 
attached. 

Note: 1) Within the code, EP, Rule, Computation Flow are used in some 
cases to refer to the same objects. 2) The eleven transformations that are applied 
15 to the user representation of the EP, after being stored in a database structure, 

are outlined beginning at line 124 of the attached source module 
"FlowManagerjava". 

E. Record Merges 

Figure 5 illustrates an example record merge. Record merging removes 
20 duplicate records from the central database 1 75. 

The following example shows how merges work and illustrates the need 
for merging duplicate records. Suppose the system 100 is using two 
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asynchronous ISMs 1 10 and 130. All outbound network traffic going through 
the proxy server 101 is routed through the firewall 103. The firewall 103 records 
the proxy server 101 as the source of all sessions passing through the proxy 
server 101 , although they originate from different workstations on the network. 
5 At the same time, the proxy server 1 01 records the destination of all sessions as 

the firewall 1.03. although their actual destinations are the different Internet 
sites. 

therefore, all sessions are logged twice by the system 100 and the 
records are skewed. The data from the firewall 1 03 indicates the destination of a 

10 given session, but not the source (see data record 520), while the data from the 

proxy server 101 records the source, but not the destination (see data record 
510). Defining a merge eliminates the duplication of records. 

A merge can be defined instructing the CEM 170 to store the destination 
data obtained from the firewall 103 and the source data from the proxy server 

15 101 in the central database 1 75. The merge will also eliminate the problem of 

skewed data by storing the correct source and destination of the session in the 
central database 175. Both network devices provide information on the URL. 
The latter can be used to identify the fact that the two seemingly independent 
records (5 1 0 and 520) are actually two logs of the same session. 

20 Two enhancement procedures are defined for the example of Figure 5. 

The trigger of the first, designated Flow One, is the Proxy Server Asynchronous 
Information Source Module. The trigger of the second, Flow Two, is the 
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Firewall Asynchronous Information Source Module. The records from Flow 
One and Flow Two are records of the same session. They both have the same 
value for the URL field. Based on this value, the CEM 1 70 identifies the two 
records are double logs of the same session. It merges the two data records 
5 taking the Source IP value from Flow One and the Destination IP from Flow 

Two as the values to be stored in the central database 1 75. 

Defining Merges 

The following describes defining merges. A merge is a set of rules that 
specify how duplicate records from multiple enhancement procedures must be 
1 0 identified and combined before being stored in the central database 175. The 

NSP can merge the records from two or more enhancement procedures. To 
define a merge, the NSP identifies the following information. 

• The enhancement procedures included in the merge. 

• How to identify duplicate records (which fields of the records must 
15 match). 

• How to combine the records; that is, for each field, which value 
(from which enhancement procedure) must be stored in the central 
database 175. (Optional) 

If the NSP does not specify how records must be combined, the records 
20 are merged as follows: 

• When the values in all but one of the fields are null, the non-null 
value is stored. 
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• When the fields contain non-null values, the value of the first record, 
received (chronologically) is stored. 

F. Additional Embodiments 

The following describes additional embodiments of the invention. 
5 In some embodiments, the user interface used by an NSP to configure 

the system 100 can be presented as a graphical representation of the data 
enhancement process. Every step in the enhancement can be shown as a block 
joined to another block (or icon or some graphical representation). The 
properties of a block define the operations within the block. In some . 
1 0 embodiments, the entire data enhancement process from network devices to the 

central database 175 can be shown by linked graphics where the properties of a 
graphic are the properties of the enhancement at that stage. 

In some embodiments, multiple CEMs 1 70 and/or central databases 1 75 
can be used as data sources (back ends) for datamart or other databases or 
1 5 applications (e.g., customer care and billing systems). 

In some embodiments, the types of databases used are not necessarily 
relational. Object databases or other databases can be used. 

In some embodiments, other platforms are used. Although the above 
description of the system 100 has been IP network focussed with Unix or 
20 Windows NT systems supporting the elements, other networks (non-IP 

networks) and computer platforms can be used. What is important is that some 
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son of processing and storing capability is available at the gatherers, the CEMs. 

the databases, and the user interface servers. 

In some embodiments, the gatherers and other elements of the system 

100. can be remotely configured, while in other embodiments, some of the 
5 elements need to be configured directly. For example, a gatherer may not be 

remotely configurable, in which case, the NSP must interface directly with the 

computer running the gatherer. 

In other embodiments, the general ideas described herein can be applied 

to other distributed data enhancement problems. For example, some 
10 embodiments of the invention could be used to perform data source extraction 

and data preparation for data warehousing applications. The gatherers would 

interface with ISMs that are designed to extract data from databases (or other 

data sources). The gatherers would perform filtering and aggregation depending 

upon the needs of the datamart (in such an embodiment, the central database and 
1 5 CEM could be replaced with/used with a datamart). The data enhancement 

would then be done before storing the information in the datamart. 

Figure 6 illustrates a system 600 where multiple systems 100 are linked 

together. This system could be an ISPs point of presence accounting system. 

The system 620 and the system 61 0 can store detailed network accounting 
20 information in their local detailed accounting databases. This information can 

then be aggregated and sent over the more expensive long distance links to the 

billing database in the system 630. Customer service information can still be 
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accessed at the detailed accounting database, but the aggregated information 
may be all that is needed to create the bills. 

G. Conclusions 

A network accounting and billing system and method has been 

5 described. In some embodiments, the system can access any network related 

information sources such as traffic statistics provided by routers and switching 
hubs as well as application server access logs. These are accumulated in a 
central database for creating auditing, accounting and billing reports. Because of 
the distributed architecture, filtering and enhancements, the system efficiently 

10 and accurately collects the network usage information for storage in a form that 

is useful for billing and accounting. 
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Appendix A - FlowManager 
Example 



5 // 

// NAME & TITLE: 

// FlowManager Analyzes Che various Enhancemnt Rules and creates Computation 
Flows 

// 

10 // ABSTRACT : 

// - Analyzes the various Enhancemnt Rules and creates Computation Flows 
// - Respond to Gatherer Requests for Instructions 

// Instructions are stored in UNIRInstruction, which can either be an element 
of a 

15 // vector or a vector, of UNIRInstructions in itself 
// - Allow for ReComputation of Particular Flows 

// - Implement Helper methods for finding interdependenci es between Enhancemnt 
Rules 

// and ISs that are referenced by them. Important when attempting to delete 
20 an IS 

// cr when an IS configuration has changed, to see which Enhancemnts mat be 
// affectetd. 

// 

// this class is closely related to the Gatherer. EnhanceCompFlowStep class, 
25. which acts 

// on the instructions created here. 

// An instruction needs to know the following: 

// 1. What type it is (recieve, enhance, send or other) 

// 2. If it is of type enhance: 



30 // 2.1 Where are the Input Unirs for the Enhancement Function 

// ("where" means position in the Arguments) 

// 2.2 What IS and what function to perform 

// 2.3 Where in the Arguments to put the function results. 



// 3. If it is of type send, needs to know to whom to send. CEM or another 
35 Gatherer. 

// 

// For further understanding, see Gatherer . EnhanceCompFlowStep 

// 

// CHANGE HISTORY: 
40 // 

// Programmer Change Date Change Description 

// --- " 

// Adi Gavish 01/12/97 Initial version 

// 

45 //** **** ***** **** *** ***** - * * 

import j ava . uti 1 . * ; 
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import Util.*;. 

import Util . Persistence . * ; 

import Util .UNI RTypes. *; 

import Util .Structs. * ; 

import Util. Log.*; 

import COM. objectspace . jgl . * i 



// ******* 

publ ic class FlowManager 

10 i 

// Current Rule we are processing 
int m_RuleID ; 

// Pipe ISM variables 
]5 boolean m_I sTarge tPipe - false ; 

int m_TargetIS - 0 ;. 

int m_TargetNode - 1 ; 

int m_TargetGatherer = 0; 

20 // Holds all Enhancments for a Rule. Alll elements are cf type. 

EnhanceOpNode . 

public. EnhanceOpSList m_0pList '= new- EnhanceOpSLis t ( ) ; 
public EnhanceOpSList mjDpTmpList = new EnhanceOpSList <) ; 

25 // All output parameters are added to this list, with their positions in 

the list 

// used to define the actual position of UNIRs in the Data Arguments 
collected by 

// the ISs of the flow. 
30 // After Output positions are defined, the input parameters are updated 

with 

// the new positions (So the Enhancement step in the Gatherer will know 
where to 

// get the input parameters of some enhancement function; 
35 // see aliocateUNIROutputData method 

SList UNIRAllocation ; 

// Temporary Storage of instructions converted from Enhancements 
SList m_TmpInst ructions ; 

40 

// HashTable For Flow Instruction Storage, access by RulelD Stores a 
complete set 

// of instructions for every Flow <== Every Rule) 
Hashtable m_Inst ructions = new HashtableO; 

45 

// Offset to fix UNIR Indexes (Take FlowlD, Step into account) 
// See PlaceHolders in Gatherer . EnhanceCompFlowStep 
final static int FLOW_OFFSET = 2; 

50 // Globol counter used in DFS Topology Sort 
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ir.t mJTimeCtr = 0 



10 



// Build All Flows 

// - Loop on all rules 

// - build an Instrucion vector for each Rule 

// - Store set of instructions per Rule in m_lnst ructions 

//*•***** "*...*.. 

public void computeFlows ( ) 



Rules RuleList = CEM. instance ( ) . getRuies ( ) , 
if (RuleList !- null) 



// Loop on all Rules 

15 for ( Enumeration El = RuleList . element s () ; E 1 . hasMor eElement s ( ) ; J 

{ 

// Compute Flow for specific Rule (If Enhancement is 

Enabled)' 

Rule CurrentRule = ( < Rule ) < El . next El ement ( ) ) >; 
20 if (( CurrentRule != null ) ( Cur rentRule . get Enabled ( ) 

=- true; j 

I 

int RulelD ■ CurrentRuie. getRuleID( ) ; 
ccmputeRuleFlows (RulelD) / 

.25 ) 
) 

) 

i 

30 — — — — * 

// Compute Flow for RulelD. At the end of this method, All 
Flowlns tructions for RulelD will be 

// generated and saved as an SList in a Hash Table, accessed by RulelD 

/ / 

35 // parameters 

// RulelD - TD of rule to build the instruction set for 

..♦**••♦♦•••••.•...•**.*.*♦*••*•**•••*"••*•****•********** ***•**•••*• 
public void computeRuleFlows ( int RulelD) 

{ ' 
40 Log. instance (> . wri teLog (new LogDebug ( "Compute Flow for Rule "+RuleID) ) ; 

m_RuleID = RulelD ; 

try 
( 

45 // l. Build Initial EnhanceOpNode set for RulelD . 

buildGUONSByP.ule (RulelD) ; 
if ( rr;_0pList . size () > 0) 
( 

// 2. Create Enhancement String Descriptions 
50' buildOpDescriptions ( ) ; 
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// 3. Convert Ail Opnums in Enhancement Input. Pa rams 



to GUOMS 

buildTranslatedOpKeyln < ) ; 

// 4. Reduce Duplicate GUONS (with same Description)' 
5 reduceSameDescriptionOps ( ) ; 

//5a. Generate OpKeysOut Information 

buildOpKeysOut ( ) ; 
// 5b. Reduce GUONS that are NOT INPUT to any other 

Non End GUON . 

]0 // Dependent on 5a. 

reduceErrorGuons ( ) ; 

// 6a. Perform Topology Sorting 
doTopolgySort ( ) ; 

// 6b. Perform "Node Coupling" Optimizations 
]5 doNodeCoupling ( ) ; 

// 7. Allocate UNIR Data Structure Key. Index 

allocateUNIROutputData < ) ; 

// 8. For every input parameter, match the appropriate 

index fronm step 7 
20 propogatelndexToInput { ) ; 

■ i ' . ■ • 

// 9. Create I ntructi'onSet 
buildlnstructionSet ( ) ; 
// 10. Save IntructionSet in Hash Table with RulelD as Key 
25 m_Instructions .put (new Integer < RulelD) , m_TmpIhst ructions ) ; 

// 11. Build Fieldlndex Vector & send to Merger (if Target is. 

DB , not pipe) 

if ( !m_IsTargetPipe) 
( 

30 buildFieldlndex (> ; 

) ■ 
Log . instance ( ) . wri'teLog ( " FlowManager . computeRuleFiows $ 

LogEvent . LCG_DEBUG, 

"Done Compute Flew for Rule M +RuleID ) ; 

35 ) 

catch; (Exception E) 
{ 

Debug. writeStackTrace ( "FlowManager .computeRuleFiows" , E) ; 

) - 

40 ) 

//•*.* **> * *********** 

// Build an initial set of EnhanceOpNodes representing all Enhancements 
per some 

45 // rule. Set some additional Info on every EnhanceOpNode . such as 

GathererlD and 

// m_IsLast Flag. 

// !!!! IT IS ASSUMED THAT Enhancements (From PERSISTENCE) is loaded 
sorted by 
50 // Field/OpNum ! ! ! 
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//* **** * 

protected void buildGUONSByRule ( int RulelD ) 

I 

m_IsTargetPipe *• false ; 
5 m_TargetNode = 1 ; 

m_TargetGatherer = 0 ; 
m_TargetIS - 0 ; 

m_OpList = new EnhanceOpSLis t (}; 
10 Nodes NodesList = CEM. instance ( ) . getNodes ( ) ; 

ISs ISsList = CEM. instance (). getISs () ; 

// Loop on All Enhancements. For Every Enhancement that matches the 

current 

15 // Rule, create an EnhanceOpNode object and add it to the OpArray. After 

// loop is done, Sort the OpArray. 
// OpArray will sort on ? 

Enhancements EnhancementList = CEM. instance ( ) . ge tEnhancements { ) ; 
Array OpArray = new ArrayO; 
20 if (EnhancementList != null) 

for (Enumeration E = EnhancementList . elements ( ) ; 

E . hasMoreElements ( ) ; ) 

{ • 
25 Enhancement Enhancement I tern = ( 

(Enhancement ME. next Element ( ) } ); 

if (Enhancementltem. getRulelD < ) RulelD) 

{ ... 
Z/printDiagnosticsPre (Enhancement) ; 
30 EnhanceOpNode EnhancementNdde- - new EnhanceOpNode ( 



Enhancement I tern ) 



\ 

35 } 



//printDiagnosticsPos t ( Enhancement Node ) , 
OpArray. add ( EnhancementNode ); 



Sort ing . sort ( OpArray ); 

// build m_OpList from OpArray 
40 if (OpArray. size < ) > 0) 

i 

String LastField = 

Rules RuleList « CEM . instance { ) . get Rules ( ) ; 
Rule CurRule - RuleList . get RuleByRul e I D ( RulelD) ; 
45 // If Rule Net Enabled, m_OpList will be empty 

if [ (CurRule != null) s & (CurRule . getEnabi ed () ) 
( 

// Get the AISM ID of this Flow 
int ISID = CurRule. getTriggerlD () ; 
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// Set Target Pipe Boolean, Target Node and other Pipe . 



variables 



m_TargetIS = CurRule . getTarge tl D ( ) ; 

if (m_TargetIS > 0) 

{ 

m_IsTargetPipe - true ; 

IS TargetIS « ISsLis t . getlSBylSID (• 



m TargetIS ) , 



if (TargetIS != null) 

10 i 

m_TargetGatherer = 

TargetIS. getGathererlD ( ) ; 

Node N = 

Nodes Li s t . get Node ByType I D ( U t i 1 . Comm . Node . NODE_TYPE_GATHERER , m_TargetGatherer ) .; 
J5 m_TargetNode = N . get Node ID { ) ; 

} 

} ' 

//Get AISM Information () 
20 IS I = iSsList .getISByISID( ISID ) ; 

if (I != null ) 
{ 

int ctr = 0 ; 

// Build First entry into m_OpList from thr 

25 Rule Header 

int GathererlD = r. getGathererlD () ; 
int ISMID = I . get ISMID ( J ; 
Node N = ' 

Nodes Li s t . ge tNodeByType I D ( Ut i 1 . Comm . Node . NODE_TYPE_GATHERER , GathererlD) ; 
30 int NodelD « N . get Node ID ( ) ; 

// Add GUON 0 with data from the Rule Header 

EnhanceOpNode OpZero = new EnhanceOpNode 

(RuleIC, f (short) 0, ISID, 0, null, null); 
35 OpZero. setGathererlD(GathererlD) ; 

OpZero. setNodelD (NodelD) ; 
OpZero, setlSMID ( ISMID) ; 

// setting GUON to the original index will garuentee 

unique value . 

40 ' OpZero . setGUON (ctr ) ; 

OpZero. setLastFlag (false) ; 
m_OpList.add ( OpZero ); 
ctr++; 

45 // Loop on OpArray, build the rest of m_OpList 

EnhanceOpNode Pr iorEnhanceCp = null 
for ( Enumeration E - OpArray . elements* ) ; 

E. hasMo re Elements ( ) ; ) 
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EnhanceOpNode CurrentEnhancement = ( 

(EnhanceOpNode) ( E . next El ement ( ) ) ); 

// Check if it belongs to The Rule 
if ( ( CurrentEnhancement != null ) 
5 (CurrentEnhancement .getRulelDi ) == RulelD ) ) 

( 

ctr++; 

// set aux data, add to list 
.1 = iSsList . getlSByXSID ( 

10 CurrentEnhancement . getlSID ( ) J; 

if (I!=null) 
{ 

GathererlD = I . getGatherer ID { ) ; 
ISMID =. I - get I SMI D( ) ; 

15 ^ ~ 

NodesList . getNodeByTypelD (Util . Comm. Node . NODE_TYPE_GATHERER, Gatherer ID) ; 

NodelD = N . getNodel D { ) ; 



20 CurrentEnhancement . setGath'ererlD (Gatherer ID) ; 

CurrentEnhancement . setNodelD (NodelD) ; 

CurrentEnhancement . set ISMID (ISMID) ; 
25 // set GUON to the original index will garuentee 

unique value . 
. CurrentEnhancement . setGUON (ctr) ; 
30 CurrentEnhancement. setLastFlag < false ) ; 

// Check if Field changed. If 

Field has changed, this means that 

// the previous EnhanceOpNode 

35 was the last for that field. 

String CurrentField = 

CurrentEnhancement . getFieldID ( ) ; 

if < < 

ICurrentField. equals (LastField) ) (PriorEnhanceOp != null) ) 

40 < 

// Field Has Changed, prior 

Op was last for a Field 

PriorEnhanceOp. setLastFlag (true) ; 
45 // Set Target Field of 

previous EnhanceOpNode 

PriorEnhanceOp. addOutFields (LastField) ; 

} 

50 
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10 



// Save for Las: Step Per Field 

Logic 

PriorEnhanceOp « 

CurrentEnhancement ; 

LastField = CurrentField ; 

// Add To List 
m_OpList.add { 

CurrentEnhancement ) ; 



Log. instance i ) . writeLog ( "FlowManager . buildGUONSByRuls" ( 
LogEvent . LOG_DEBUG , Cur rent Enhancement . toSt ring ( ) ); 

15 ■ ' 

else 
{ 

Log . instance (). writeLog (new 
LcgError ("buildGUONSByRule: No IS found for "^CurrentEnhancement . ge tISID ( ) )) ; 

20 . > 

) . 

i 

// End Of Loop Logic. Same as if Field has 

changed. 

25 if (PriorEnhanceOp != null) 

( 

PriorEnhanceOp. setLastFlag < true) ; 
PriorEnhanceOp . addOutFields (LastField) ; 



30 



j3 //** 

// Print Diagnostics on the passed Enhancement. 
// Currently Not Used, 

//**** ************ *********** ******* ♦*** ******** 

protected void print Diagnostics Pre ( Enhancement Enhancement I tern) 

40 ( 

try 
{ 

byte{] buf = Enhancement I tern . getOpKeyln () ; 
String. InputList = "[" ; 
45 Arguments a = new Arguments ( buf ); 

for (Enumeration EK = a . elements () ; EK. hasMoreElements ( ) ; ) 

i 

short value = 
( (UNIRShort ) SK. next Element { ) ) . getShort Value ( ) ; 
5Q InputList +- Short. toString (value) + '\ "; 
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value = 
{ (UNIRShcrt) EK . nextEiement ( ) ) . getShortValue ( ) ; 

InputList — Short . toString (value )+" , " ; 

.j 

5 inputList +="]"; 

Log . instance ( ) . writeLog ( "FlowManager . bui ldGUONSByRule M , 
LogEvent . LOG_DEBUG, "Persistence! tern: 

InputList) ; 

10 ) 

catch (Exception e) 

{ 

} 



15 



// Print Diagnostics on the passed EnhanceOpNode. 
// Currently Not Used. 

//*••*•****•* * 

20 protected void printDiagnos ticsPos t { EnhanceOpNode. 

EnhancementNode ) 
{ 

try 

( 

25 byte [] buf = EnhancementNode . getOpKey In {.) ; 

String InputList = " ( " ; 
Arguments a =*. new Arguments ( buf ) ; 

for (Enumeration EK = a . el ements ( ) ; EK . hasMoreEl ements ( ) ; ) 
1 

30 short value = 

( iUNIRShcrt) EK . next Element ()). getShort Value () ; 

• InputList +» Short . toString (value) +" , "; 
value 

( ( UMIH.Short ) EK . next Element ( ) ) .ge t Short Val ue ( ) ; 
35 InputList +- Short . toString (value) +" , 

) 

InputList += "] n ; 

Log . instance^ ) . writeLog ( "FlowManager . bui ldGUONSByRule " , 
LogEvent . LOG_DEBUG, "Struct 1: 

40 " + Input List) ; 

InputList = " [ " ; 
for i Enumeration EK = 
EnhancementNode . ge t Pa rsedOpKeyln (}. elements {) ; EK . hasMoreElement s ( •;• ; ) 
45 • 

inc value = { (Integer) EK. next Element ( ) j.intValueO. 
InputList += Integer . toString ( value )+" , "; 
value - ( (Integer) EK. next Element ( ) J.intValueO 

50 InputList += Integer . toString (value) , "; 
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} 

InputList += "J"' 

Log. instance*) .writeLog < "FlcwManager . buildGUONSByRule" 
LogEvent . LOG_DEBUG, "Struct 2: 



'-•-InputList) , 



} 

catch ( Exception e) 

10 i 

) 



15 // ABSTRACT 

// convert the m_OpList to an Instruction Set the Gatherer Can 

Understand 

// (a set of UNIRInstructions) 

// 

20 // Description 

// Translate m^OpList to sets of Enhancement Instructions. Divide the 

Enhancements 

// by Different Gatherer sequences, wrap each segment with a Receive 
Instruction 
25 //". and a Send Instruction. 

..••..***..**•*♦*•••*******?**•***.**•******•* 

//.•.♦•***•••.-.•.••*♦♦***•*******•*•**************** 

// Helper method to add an instruction of type Send 

30 //• — * — — — — . 

void AddSendStepdnt StepCtr, int NodelD, int ISID, int 

ISMID, int GathererlD) 

( 

UNIRInstruction u = new UNIRInstruction <m_RuleID, 

35 

StepCtr, 

UNIRInstruction. INSTRUCTIONS END, 

NodelD, 

4 q 0, ISID, ISMID, 

null , . null , 

GathererlD ) ; 

Log.instancen . writeLog ( "FlowManager . bui ldlnstructionSet " 

LogEvent . LOG_DE3UG, 

"Adding Instruction "+u> ; 



m 



Tmplnst ructions .pushBack (u) ; 



1 



50 // * 
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// Helper mechod to add an receive of type Receive. 

//**** *** *»***♦-** * * * 

void AddReceiveStep (int StepCtr, int NodelD, int" ISID, int 

ISMID, int GathererlD) 

5 ( 

UNIRInstruction u = new UNIRIns truction (m_RuIeID, 
StepCtr, 

10 UNIRInstruction. INSTRUCTICN_RECEIVE, 

NodelD, 

0, ISID, ISMID, 
null, null, 

GathererlD ); 

]5 Log. instance ( ) . wri teLog ( " FlowManager . bui ldlns t ructionSet " , 

LogEvent . LCG_DEBUG, 

"Adding Instruction "+u); 
mJTmpInstructions .pushBack (u) ; 

20 :■ 



// Helper method to add a receive of. type Enhance 

//*»*«*■*** ...„***»»** - *»***»,** 

25 void AddEnhanceStepUnt StepCtr, int NodelD, int ISID, int 

ISMID, inc. GathererlD, 

int FunctionlD, byte ( ] . inParam, byte() 

outParam) 

{ 

30 UNIRInstruction u = new UNIRInstruction <m_RuleID, 

•StepCtr, 

UNIRIns t ruction . INSTRUCT I OM_ENHANCE, 
35 NodelD, 

FunctionlD, ISID, 

ISMID, 

inParam, outParam. GathererlD); 
40 Log. instance ( ) . wri teLog ("FlowManager . buildlnstructionSet " , 

.Log Event . LOG_ DEBUG, 

"Adding Instruction "+u) ; 
mJTmpInstructions . pushBack (u) ; 

45 ) 



//**,.,♦**,».*»*************** **' ' 

// Helper method to add an receive of type Pipe Give 
,/******.*♦*+♦*************.*»********************** 
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void AddGiveStepiint StepCtr, int NodelD, ir.c ISID, ir.t 

ISMID. in: GathererlD, 

int FunctionID, byte[] inParam, byte ( ] 

• outParam) 

5 i 

UNIRlnstruction u « new UNIRlnst ruction <m__Ruleip, 
StepCtr, 

10 UNIRlnstruction. INSTRUCTION_GI VE , 
ISMID, 



20 



NodelD, 

FunctionID, ISID, 



J5 inParam, outParam, GathererlD); 

Log. instance () . wri teLog (" FlowManager . buildlnstructionSet'" 



LogEvent. LOG_DEBUG, 



"Adding Instruction 7+u) ; 
m Tmplnstructions . pushBack (u) ; 



* * * 

25 protected void bui idlns tructionSet { ) 

{ 

Log. instance () . wri teLog ( "FlowManager . buildlnstructionSet " , 
LogEvent . LOG_DEBUG, 

"Build Instruction Set for Rule 

30 M +m_RuleID) ; 

EnhanceOpNode PriorOp = null ; 
int PriorNodelD = -I; 
m_TmpInstructions - new SList (); 
int StepCtr = 0; 

35 for (SListlterator I = m_OpList . begin () ; ( ! I . a t End ( > ) ; I . advance! ) ) 

{ 

EnhanceOpNode Opl - (EnhanceOpNode) r. get () ; 

if ( ! opl . getLastFlag { ) ) 

I 

40 if (Opl.getNodelDt ) !» PriorNodelD) 

I 

// Add Send Instruction to close prior node 

sequence 

if (PriorNodelD != -1) 

45 i 

AddSendStep (StepCtr , 

Opl . getNodelD ( ) , Opl . getlSlD ( ) , 

Opl .get ISMID () , 

PriorOp . ge t Gatherer ID ( ) ) ; 
50 StepCtr++; 
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sequence 
5 Dpi . getlSIDO , 

Opl . getGatherer ID ( ) ) ; 

10 )■ 



20 



// Add Recieve Instruction to start current node 
AddReceiveStep (StepCtr, Gpi . getNodeip ( ) , 

Opl .getlSMIDO , 

StepCtr++; 
PriorNodelD = Opl . get Node I DO; 



PriorOp • Opl ; 
// Add Enhancement Step 

AddEnhanceStep (StepCtr, Opl . gecNodelD ( ) , 

Opl .getlSIDI ) , Opl.getlSMIDM , 
\^ Opl . getGathererID( ) , 

Opl . get FunctionID ( ) , 



Opl .getOpKeylnAsArray ( ) , Opl . getOpKeyOutAsAr ray ( ) ) , 

StepCtr++; 



// The Last Step, Add Send Instruction to CEM 

// Check globol Pipe flag. If this Flow Target is some Pipe ISM, 
// Set the Instructione Type to I NSTRUCTION'_GI VE and set correct Node 
25 if (PriorOp !=. null) 

{ • 

if (m_IsTargetPipe) 

{ 

// if target is" in a different node, add a send and recieve 
30 if (m_TargetNode != PriorOp : getNodeID< } ) 

{ . ■ 

AddSendStep (StepCtr , m_Tar ge tNode , 0,, 0, 

PriorOp. getGatherer ID { ) ) ; 

StepCtr++; 

35 AddReceiveStep(StepCtr, 0, 0, 0, 

m_TargetGatherer) ; 

StepCtr++; 

} 

// adding the final Give instruction. Dont forget Binding 

40 information 

byte [ ] bmdinglnfo = 

buildPl SMBinding Information () ; 

AddGiveStep (StepCtr , m__Ta rgetNode , m_TargetIS, C, 
mJTargetGatherer , 0, 

45 bindinglnro, null); 

} 

else 

i 

// Send Data to CEM 
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AddSendStep (StepCtr , Util . Comm. Node . MODE_T Y PE_C EM , 
0, 0, PriorOp.getGathererID( ) ) ; 

i 

StepCtr++; 

5 ) 

) • " ■ • 

//*•• ************ * ******* + + + + ********* 

// doTopolgySort ( ) 

' 10 // m_OpList can be seen as a Directed Acyclic Graph (DAG), and an 

algorithm 

// (introduction To Algorithms, Chapter 23) to sort the graph is applied. 

The 

// result of this procedure is a list where the order of operations is 

15 correct, 

// meaning that all enhancments whose ouput is used as input happen 
before the 

//enhancments that use them. . 
//****,* **** ...•♦•*•..*■*.♦..***.•*••••*•••■ ************ 

20 



45 



50 



protected void vis i tDFS ( EnhanceOpNode Op) 
( 

Op. setCplor (EnhanceOpNode . GRAY) ; 
Op. setDFSDiscoverTime (m_TimeCtr ) ; 
25 m_TimeCtr++; 

// For Each Adjacent Vertex (All Output GUONS) 
for (Enumeration AdjVert Op . getOutputGUONs ( ) ; 
AdjVert . hasMoreElements () ; ) 

I 

30 int OutGuon = 

(( Integer ) AdjVert . next Element ()). intValue () ; 

EnhanceOpNode OutOp - m_OpList . findGuon (OutGuon) ; 
// Explore Edges Each ( Guon , OutGuon ) pair is an 

edge 

35 if ( ( outOp != null) &s (OutOp. getColcr <! == 

EnhanceOpNode. WHITE) ) 

( 

visitDFS (OutOp) ; 

OutOp. setPredecessor (Op) ; 

40 ) 

} 

Op . setColor (EnhanceOpNode . BLACK) ; 

Op.setDFSFinishTime (m_TimeCtr) ; 
m TimeCtr++; 



m OpTmpList . pushFront (Op) ; 



//- 
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protected void doTopolgySort ( ) 

// Create Temprary List to Store Ordered Ops. 
rr^OpTmpList = new EnhanceOpSLis t ()/ 

5 

m_TimeCtr = 0 ; 
// For each vertex 
for (int i=0; i<m_OpList .size () ; 

! 

10 EnhanceOpNode Opl = ( EnhanceOpNode ) m_OpLis t . a t ( i ) ; 

// if not used in DFS yet (Color White) 

if ( Opl . getColor ( ) =- EnhanceOpNode . WHITE ) 

visitDFS ( Opl ) ; 

15 ) 

} 

// Temporary List is now the correct one, move it. to normal list 
-:_OpList - new EnhanceOpSLis t ( m_OpTmpList ) ; 

20 

// ******* ********** *•*****♦.*******************• ***♦ 

// doNodeCoupling ( ) 

25 // Node Coupling attempts to move enhancments that share the same 

gatherer closer 

//in sequence, as a way to reduce traffic between gatherers. This is 
done without 

// damaging the basic order of the enhancements established earlier in 

30 the 

// Topology Sort. 

,/************** * * ***** ************* ************** - 



35 — 

// Find if inserting Op at Pos will cause sort order 

conflict 

protected boolean noHarmToSort ( EnhanceOpSLis t OpList, int 

pos, EnhanceOpNode Op) 

40 ( 

for (int i=pos; KO'pList . size ( ) ; i + +) 

( . • 
EnhanceOpNode OpToCompare = 

( EnhanceOpNode) OpList. at (i) ; 
45 // Check if Op has input that uses Output 

of OpToCompare 

// Loop on all inputs of Op, see if the Guon value 

matches OpToCompare . Guon 

for (Enumeration EK - Op . ge t Inl tera tor ( ) ; 

50 EK. has Mo re Elements ( ) ; ) 
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i 



(OpInpunParam) EK. next Element ( ) ; 
OpToCompare . getGUON () } 



OplnputParam inParam = 

if ( inParam. getGUON ( ) 



{ 



// No insertion Possible 
return false ; 

} 

} 

} 

// No Conflict Found, Insertion Allowed 
return true ; 



15 



20 



OpList 

LnhanceOpNode Cp ; 



25 



/v 

// find last position of first same node sequence in 
protected int f indSameNodePos ( EnhanceOpSLi s t OpList, 

i 

int Node ID - Qp.getNodelDM ; 
boolean StartSeqFound » false ; 
int ctr = 0 ; 
int ResultVal - -1; 
for (SListlterator I = OpList . begin (); ( ! I . at End ( ) j ; 



I . advance ( ) ) 



30 { EnhanceOpNode ) I . get ( ) ; 



35 



40 



what weere looking for 



EnhanceOpNode OpToCompare = 

if (NodelD == OpToCompare .getNodelD ( ) ) 
( 

StartSeqFound = true ; 

! 

else 
< 

if ( StartSeqFound ) 
( 

ResultVal = ctr 

break; // get out of loop, we found 



ctr++ ; 



45 



return ResultVal 



50 



57 



SUBSTITUTE SHEET (RULE 26) 



WO 99/27556 



PCT/US98/24963 



protected void doNodeCoupiing ( ) 

// Create Temprary List to Store Ordered Ops. 
5 m_OpTmpList = new EnhanceOpSList U; 

for (SListlterator I = m_OpLis t . begin O ; (II. atEnd ( ) ) ; I .advance!) ) 

i 

EnhanceOpNode Opl « ( EnhanceOpNode ) I . get {) ; 
10 // Add Opl to the Temporary List, either at the end (No Node. 

Optimization Possible) 

//or Insert in the middle (Node Optimization, Couple Nodes) 
int nl * f indSameNodePos (m_OpTmpLis t , Opl); 
if (nl -1) 

15 (. 

// No Node Optimizations possible, add to end 
mJDpTmpList. pushBack (Opl ) ; 

} . 

else 

20 < 

if < noHarmToSort (m_OpTmpList, nl, Opl) ) 
( 

// Insertion OK, insert to group same node Ops 

• together 

25 m_OpTmpList . insert (nl, Opl); 

) 

else 

.1 



30 cannot inset, add to end 



35 



// insertion would hurt the input - >output , order , 
m_OpTmpList .pushBack (Opl ) ; 



// Temporary List is now the correct one, move it to normal list 
m_OpList = new EnhanceOpSList < m_OpTmpList ) ; 

) 



40 //* 



// Create an image of the Data to be produced by this Rule. This 
allocates ouput 

// positions for every Enhancement. These positions are used later to 
45 update the 

/ / input params . 

//-****.***» ..*.♦•,»*♦♦***»*************** *- * • - * *** 



protected void al loca teUNIROu tputData ( ) 

50 i 
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15 



20 



25 



30 



UN I RA1 location « new SListO; 
// For Each OP 
fcr (int i«0 ; i<m_OpList . size ( ) ; i++ ) 

EnhanceOpNode Opl = (EnhanceOpNode ) m_OpLis t . a t ( i ) ; 
if {Opl != null) 



// For Each Output Parameter 
for (Enumeration EK = Opl . getOutputKeys O ; 



10 EK . hasMoreElements O ; ) 

{ 

is key, 2nd is guon 
Integer (Opl .getGUON O ) ); 



integer Key = (Integer) EK. next Element ( ) ; // first 
GuonKeyParam NewKey = new GuonKeyParam ( Key , new 
if (NewKey ! -null J 

UNIRAllocation. pushBack (NewKey) ; 

) . '. 



) 



// propogatelndexToInput () 

// Based on the Unir Allocation, match every input parameter, with 



the 



index in the 

// UNIRAllocation. 



35 



jrotected int findlndexinAl location ( OptnputParam Kevin) 

( ■ 

GuonKeyParam CurlnKey = Keyln.getGuonKey.O-; 
for { int 1=0; i<UNIRAllocat ion . size () ; i + + ) 



40 (GuonKeyParam) UNIRAllocation. at ( i ) ; 



GuonKeyParam KeylnAllocat ion 



if (CurlnKey . equals (KeylnAllocat ion) ) 
I 

return i . ; 



45 



return -1 ; 



50 



//--- 
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protected vc:c propogatelndexTolnput ( ) 
// For Each OP 

for (SListlterator I = mJDpList . begin () ; { ! I . at End ( ) ) ; I . advance () ) 
5 i 

EnhanceOpNode Opl * ( EnhanceOpNode ) I . get ( ) ; 
// For Each Input Parameter 

for (Enumeration EK = Opl . get Inl terator { ) ; EK . has Mo re Elements ( ) ; 

10 i 

OplnputParam inParam = f Oplnput Par am) EK . nextEl ement ( ) ; 
//Get index of pair in UNIRAllocat ion 
int n = f indlndexinAllocat.ion < InParam) ; 
if (n == -1). 

15 ( 

// TBD, Error 

) 

InParam. setlndex<n + FLOWJDFFSET ); 

}. 

20 Log . instance ( ) . writeLog ( "FlowManage r . propogat elndexToTnput " , 

LogEver.t . LCG_DEBUG, 

"Propagated Input Op: " +Opl); 



25 



// ** ** ............... 

// For every Enhancement 

// Build List of Enhancments that need the output of some enhancement 
30 as Input. 

// 

. // This information is needed later to weed cut Non productive 
enhancements and 

// to perform the Topological sort on m_OpList 

35 // 



protected void bui IdOpKeysOut { ) 
( 

// Opl Loop : Loop on All Enhancements in m_OpList 
40 for (SListlterator I » m_OpList . begin {) ; ( ! I . at End ( ) ) ; I. advance O) 

{ 

EnhanceOpNode Opl = (EnhanceOpNode )I . get () ; 
// Op2 Loop : Loop on All Enhancements in m__OpList 
// Compare all Enhancments to Opl (Except itself) 
45 for (SListlterator J = m_OpList . begin () ; ( ! J . at End { ) ) ; 

J. advance ( ) ) 

{ 

EnhanceOpNode Op2 = { EnhanceOpNode ) J . get () ; 
// No need to compare Op to itself 
50 if (0pl!=0p2) 
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EK. hasMoreElements ( ) ; ) 
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// If Opl participates as input in Op2, add Op2 to 
// Enhancements that need Opl output. This list' is 

for (Enumeration EK « 0p2 . get inlterator ( ) ; 

[ 

OpInputParam InParam = 



10 (OpInputParam) EK. nextElement O ; 

IU P • if (inParam.getGOOHO — Opl-.getGUONO) 



15 



Opl . addOutputParam ( InParam. getKey ( ) , Op2 . getGUON ( ) ) ; 

} 

) 

) 

} 



it* 



// Reduce Duplicates. No need for duplication of process if .another 

25 enhancement , . , 

„ already creates the needed Output. Keep one, of the duplicates, delete 

0th 7/' Pass thru all input keys of all enhancments, replacing, any reference 
to the 

30 // Deleted GUON with the Surviving GUON. 



protected void reduceSameDescr iptionOps ( ) 

i .' 
35 for (int i=0;i<m_OpList.size() 

EnhanceOpNode Opl = (EnhanceOpNode ) m_OpLis t . at ( i ) ; 

// Move from end to Current Pos because we might delete an object 
for (int j=m_OpList.5i2e<) -l;j>i; j — ) 

40 < 

EnhanceOpNode 0 P 2 = ( EnhanceOpNode >m_OpList.at A ? ) ; 

if < Opl. getDescriptionO .equals* Cp2 . getDescr iption ( ) ) 

( ' 
45 L og. instance () . writeLog ( "FlowManager . reduceSameDescriptionOps " , 

LoqEvent . LOG_DEBUG, 



50 



"Reducing Description : 
•+Op2 . cetFieldlDf ) + " : " A 
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Op2 . getDescription ( ) ); 

// Merge outfields of deleted guon with surviving 

guon 

5 Opl .addOutFields <0p2 . getOutFields ( ) ); 

int newGUON = Opl . getGUON ( ) ; 
int oldGUON Op2 . getGUON ( ) ; 
m_OpList . remove (j ) ; 
// GUON Replacement Pass 
jO for (int k=0; k<m_OpList .size ( ) ; k++) 

{ 

EnhanceOpNode Op3 = 

< EnhanceOpNode )m_OpList . at <k) ; 

Op3 . SwapInputKeyGUON (oldGUON, newGUON) ; 

15 > 

//Mark for Garbage Collection 
Op2 - null ; 



20 ) 



30 



25 // Reduce Error Guons, that do not produce an output needed as input by 

any other 

// guon. They can be identified if they are not "Last" and the output 



list is empty. 

//***• 



protected void reduceEr rorGuons ( ) 

{ 

if (m_OpList . size ( ) > 0} 
{ 

35 for (int i«m_OpList.size ( ) -l;i>-0.; i — J 

{ 

EnhanceOpNode Opl = (EnhanceOpNode ) m_OpLi s t . at { i ) ; 
// If Not Last Guon per some field. . . 
if ( !Opl .getLastFlag ( ) ) 

40 « 

// Check if any Outputs are defined 
Enumeration ChkOutput = 

Opl . getOutputGUONKeys { ) ; 

if ( \ ChkOutput . hasMoreElements () ) 

45 < 

Log. instance ( ) . wri teLog (" Fl owManager . reduceEr rorGuons " 

LogEvent . LOG_DEBUG, 

"Reducing Bad Enhancment 
: "+Opl . getGUON ( ) + " : "+Opl . getDescription ( ) ) ; 
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// No Output Elements for this 

m_OpList . remove ( i) ; 

// Mark for Garbage Collection 

Opl - null ; ... 

} 



10 



15 



20 



25 



30 



// Build Translated OpKeyln for every Enhancment 

//!!!! IT IS ASSUMED THAT Enhancements (From PERSISTENCE) are loaded 



sorted by 

// Field/Opnum ! ! ! 
//* ♦*.»**** 



protected void bui ldTrans latedOpKeyln ( ) 

for (SListlterator I = m.OpList .begin ( ) ; { ! I . atEnd ( ) ) ; I . advance () ) 

{ . • ' . 

EnhanceOpNode Op = ( EnhanceOpNode ) I . get ( ) ; 

if (Op != null) 

( 

Op. translateOpKeyIn<m_OpList ) ; 

Log. instance () . writeLog < "FlowManager . buildTranslatedOpKeyln" , 
LogEvent . LOG_DEBUG, Op . toString ( ) ); 



} 



35 



40 



45 



50 



// Build a Description for every Enhancment 

// mm IT IS ASSUMED THAT Enhancements (From PERSISTENCE) is loaded 

sorted by 

// Field/Opnum ! ! ! ■ 

//*••* ***** ********** ****** 



protected void bui IdOpDescript ions ( ) 

for (SListlterator I = m_0 P List . begin () ; ( ! I . atEnd ( ) ) ; I . advance 

EnhanceOpNode Op = ( EnhanceOpNode) I . get () ; 

if (Op != null) 

( 

String S = get Descr ipt ion ( Op ); 
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Leg. instance () . writeLog r FlowManager . buildOpDescriptions", 
LogEvent . LOG_DEBUG, 

"Op Description : 

5 »+Op.getFieldID()+ M : M +Op.getGUON(). + ": "+S); 
} 

) 

} 

10 

// get Textual representation of the first EnhanceOpNode Input Key 



15 -protected String getKeyText (EnhanceOpNode Op) 

{ 

String ResultVal - ; 
try 

i 

20 • if (Op != null) 

I 

Arguments args = new Arguments ( Op . getOpKeyln ( ) >. 
int n - 

( (uNIRShort ) args ; get (ArgKeys . KEY__KEY_ID) ) . getShor tValue ( ) 

ResultVal ■ Integer . toString <n) ; 

} 

) • 
catch (Exception e) ( ) 
return ResultVal ; 

30 ) 



// Build a Textual description of a regular EnhanceOpNode Op 

35 // 

// the methods "parseArgs" and "getDescription" are used in a recursive 



. manner 



40 protected String parseArgs (EnhanceOpNode Op) 

i 

String s = , "* ; 

try 

t 

45 // Loop On All Op Key In 

Arguments a = new Arguments ( Op . getOpKeyln ( ) >; 

for (Enumeration EK = a . elements () ; EK . hasMoreElements ( ) ; ) 

{ 

// Add Description of Input Param OpNum 
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short Opvalue = ( (UNIRShort) 

. EK.nextElemen.t()>.getShortValue(); 

EnhanceOpNode CurrentOp -. nv_OpList . f mdOp (OpValue, 

Op . getFieldID( ) ) ; 
- if ( iop.equals(CurrentOp) ) 

{ 

s +- getDescription (CurrentOp) ' 
// Add Description- of Input Param Key 
short KeyValue « ((UNIRShort) 

10 cK.nextBlementO) .getShortvalueO; 

s integer. toSt ring (KeyValue ) ; . 

//If More Param, delimit by comma , 

if ( EK.hasMoreElements () ) 



15 



20 



30 



35 



■ { 

s+='V 

} 

) 

else 
{ 



tog.inscanceO.writeX-ogCTlowManager.parseArgs^ Log Event . LOG_ERROR, 

"Corruotion Detected in Data (Illegal Recursion) " 

throw ( new Exception () >; 

25 » 
) 

catch (Exception EA) 
s = 



return s ; 



//••••*•** 

'// Build a Textual description of the EnhanceOpNode Op 

7/ the methods "parsers" and "getDescription" are used in a recursive 



manner 

40 //• 



protected String get Description ( EnhanceOpNode Op) 
< 



String s = op . getDescription () ; 
.45 // if no description was built ye 



ilt yet, build it and save for next 



if ( s==null ) 



// if last Step for some field, just parse 
if (Op.getLastFlag () ) 
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s ■ parseArgs (Op) , 



25 



50 



) 

else 



5 if (Op. getOpNum( > ==0) 

{ 

// First Op, has no function 

s = Op.getlSIDO + " . " + getKeyText <Op) ; 

} 

10 else 

( 

// Function, Must show Func ID + Input 



Parameters . 



s - Op.getlSIDO + * + Op . get FunctionID ( ) 



15 + ("; 



s += parseArgs (Op) , 
s +- ")" ; 

) 

// Save for next time 

20 Op . setpescription (s ) ; 

// 

} 

return s ; 



// Scan All Rules, Loop On All Instructions, send matching instructions 
to Gatherer 



public void set Inst ructionsByGatherer ( int GathererlD, Gathere rController GC 

) 

{ 

35. Rules RuleList = CEM. instance <) . getRules () ; 

if (RuleList != null) 

{ 

// For Each Rule 

for (Enumeration El = RuleList . elements () ; El . hasMoreElements ( ) ; ) 

40 { 

// Build Flow for specific Rule 
Rule CurrentRule = ( (Rule ) ( El . nextElement ( ) ) ); 
if {( CurrentRule != null ) && (CurrentRule . getEnabled ( ) 



true) ) 

45 ( 



int RulelD = CurrentRule . getRulelD () ; 

setlnstructionsByRuleByGatherer < GathererlD, 



RulelD, GC ) ; 
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// * ************ ** 

// Send All Instructions per Rule/ -per Gatherer to the Gatherer 
// Thru the Ga thererCcnt rol ler 



10 synchronized public void setlnst ruct ionsByRuleByGa therer { int GathererlD, 

int RulelD, GathererController GC ) 

< . 

SList Tmplnstructions = < SList ) m_Inst ruct ions . get (new 

Integer (RulelD) ) ; 

15 vector v = new Vector!) ; // Hold UHIRIns truction belonging to 

some Gatherer 

. // Loop on ATI instructions of a Rule, Add Gatherer • Matches to v. 

Send v to GC 

for (Enumeration EOps = Tmplnstructions . elements ( ); 
20 EGps . hasMofeEiements ( ) ; ) 

{ • . .. 

UNIRIns truction Op = ( UNIRInst ruction) ECps . nextElement () ; 
if (Op. getGathererlD ( ) === GathererlD) 
{ 

25 v.addElement (Op) ; 

) 

). 

try 
1 

30 

Log . instance ( ) . writeLog ( "FlowManager, set Ins t ruct i onsByRuleByGa therer " , 
LogEvent . LOG_DEBUG, "Disable Flow "+RuleID+ " on " + GathererlD); 

GC . disableFlow < RulelD) ; 
if (v. size ( ) > 0 ) 

35 ( 

Log . instance ( ) . writeLog ("FlowManager . set Instructions By RuleByGa therer " , 
LogEvent . LOG_DEBUG, "Upload Flow "+RuleID+ " on " + GathererlD); 

GC. sendFlowData (RulelD, new UNI RI nstruction ( v) ); 

40 

Log. instance ( ) . writeLog ( "FlowManager . set InstructionsByRuleByGatherer" , 
LogEvent . LOG^DEBUG, "Enable Flow "+RuleID+ " en " + GathererlD); 

GC : enableFlow(RulelD) ; 

) 

45 } 

catch ( Exception e) 

{ 

1 

! 
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// Pack the Binding Information between current Flow to the input 

// parameters of the target Pipe ISM. 

// This information is added to the Give instruction and sent to the 

// Gatherer where it is used to pass data from the output UNIR of 

// the current flow to the input parameters of the target flow. 

// RETURNS 

// a byte array containing Binding information in the following format: 

// pairs of shorts made of the index of the data in the output UNIR, 



followed 



// by the key value of the target input parameter. 
//*♦***** ********************************* 



byte [ ] buildPISMBindinglnf ormation ( ) 
I 

try 

{ 

Arguments args = new Arguments () ; 
Vector v = new Vector (); 
// Loop on All Enhancements, 
for (SListI terator I = m_OpList . begin () ; ( ! I . at End ( ) ) ; 



I . advance ( ) ) 



Output Unir 
Output Unir 
Input Parameters 

flow data and put 



Key 



EnhanceOpNode Op = ( EnhanceOpNode ) I . get ( ) ; 
if (Op null) 
{ 

int Index = 0; // Position of data in 
short Key - - 0; // Key ID of data in 
int inKey =0; // Key ID of data in 

// the binding process will 

// "take the UNIR in the Index position from the 

// in the PISM input parameter . in inKey" 

// One to many relation: One input to Many output 
// DEFINE INPUT : Get Index & key of first input 

Enumeration EK = Op . get Inl terator () ; 
if ( EK. hasMoreElements { ) ) 



i 



(Op Input Pa ram) EK . next Element ( ) ; 



OpInputParam InParam 



Index = InParam. m_KeyIhdex ; 
Key = (short ) InParam. m_KeyID; 
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// DEFINE OUTPUT (S) Loop on all Out Values 
for (Enumeration EFlds - 
Op. getOutFields () .elements ( ); EFlds . hasMoreElements (); ) 

( 

5 String FieldID ■= 

(String) EFlds . next Element () ; 

Log. instance () . wri teLog ( '* FlowManager .buildPI SMBinding Information", 
LogEver.t . LOG_INFO, 

jq "Got Pipe 

Binding: "+ FieldID + •' ; "+ Index +" ; H + Key) ; 

// in PISM, FieldID is a String 

Representation of the KeylD 

// defined as string for 

15 compatabiliy with Fields (see bui ldFieldlndex ) 

inKey = Integer .valueOf ( FieldID 

) . intValue ( ) ; 

// Add to Arguments; 

20 

args.put ( ArgKeys . ARGKEY_ENHANCEMENT_OP_NUM, new UNI RShort ( ( short ) Index) ) ; 

args . put ( ArgKeys . KEY_KEY_ID, new 

UNIRShort ( (short) inKey) ) ; 

25 » 

} 

i 

return args . toByte Array ( J ; 

) 

30 catch ( Except ion e) 

t 

Log . instance () .wri teLog ( M FlowManager . bui ldPISMBindinglnf ormation " , 
Log Event . LOG_ERROR, 



35 



40 



"Error in 

Binding Info, " + e) ; 

Debug. writeStackTrace ("FlowManager. buildPISMBindinglnf ormation" , e); 



} 

return null 

} 



♦ 

// Builds a vector of all DB fields that participate in a specific Flow. 
// (The current flow, in m_OpList) 

// This list is built of FieldlndexStruct elements and sent to the 

50 Merger . 
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// The Merger uses this information in processing incoming UNIRs from 

this 

// flow. 

// The vector is a cross reference between the DB FieldID and the 
5 position 

// the data to be put in the field from the incoming UNIR for this flow. 

// 

void buildFieldlndex { ) 

10 < 

Vector v = new Vector <); 
// Loop on All Enhancements 
for (SListlterator I = m_OpList . begin () ; ( ! I . at End ( ) ) ; I .advance!).) 

{ 

15 EnhanceOpNode Op = < EnhanceOpNode > I . get () ; 

if (Op != null) 

int Index » 0 ; 
short Key = 0; 

20 Enumeration EK = Op. getlnl terator <.) ; 

// Get Index & key of first input key 

if ( EK. hasMoreElements ( ) ) 

< 

OpInputParam InParam = 
25 (OpInputParam) EK . next Element (■) ; 

Index = InParam. m_KeyIndex ; 
Key = (short ) InParam. m_KeyID; 

} 

30 // Loop on all Fields Out, add all fields in list to 

vector with 

. // Index & ket stored earlier. 

for (Enumeration EFlds = 
Op. getOutFields (). elements () ; EFlds . hasMoreElements (); ) 

35 ( 

String FieldID - 

(String)EFlds. next Element ( ) ; 

v. addElement (new Fie ldlndexS t ruct (FieldID, 

Index, Key) ) ; 

40 > 

} ■ 

) 

if (v.sizeO > 0) 

{. 

45 // Call Merger 

Merger m = CEM . ins tance ( ) . getMerger ( ) ; 
Log . instance!) . writeLog (new LogDebug < "Vector: \n M +v) ) ; 
m. addRule (m_RuleID, v) ; 

) 

50 else 
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// Call Merger 

Merger m = CEM. instance li . getMerger O ; 
Log. instance (i . wr i teLog ( new LogDebug ( "De le t ing Ru 1 e " +m__Ru 1 e I D ) ) , 
m.deleteRule <m_RuleID) ; 

) 



// Get Ail Enhancraents wich macching FieldID 
// Return Vector cf affrected Rules 
//* * **** 



1 5 public vector getRulesWi thField { String FieldID) 

t 

Log. instance ( ) . wri teLog ( "FlowManager . ge t RulesWi thRuleField" , 
LogEvent . LCG_DEBUG, "Get Enhancements with Field "^FieldID); 
vector ResultVal = new VectorO ; 
20 OrderedSet Af f ectedRules = new OrderedSet (false ) ; 

Vector ToDel = new VectorO; 
Enhancements EnhancementLisc = CEM . instance ( ) . get Enhancements t J ; 
for (Enumeration E = Enhancement Li s t . el ements () ; 
E . hasMoreElements ( ) ; ) 

25 i 

Enhancement CurrentEnhancement « < 

( Enhancement ! ( E . nextElement ( ) ) ); 

if ( ( FieldID. equals ( Cur rentEnhancement . qe t FieldID ( ) ) > ) 

30 Aff ectedRules - add ( new Integer » 

Cur rentEnhancement . getP.u lei D ( } ) ); 



35 // Set Result Val 

for (Enumeration AR = AffectedRuies .elements () ; 

AR. hasMoreElements () ; ) 

{ 

Integer P.ulelDObj = ( Integer ) AR . next Element ( ) ; 
4Q ResultVal. addElement (RulelDCbj) ; // add Rule to Result Set 

i 

return ResuitVal ; 



45 //• 



// Delete All Enhancments with matching FieldID. RulelD from List 
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public void deieteRulesWithField (Vector F.uieList. £:r:nc FieidlDI 

for (int i=0; i<RuleList. size ( ) ; i + 

5.1 

int RulelD = 
( (Integer) ( RuleLis t . element At (i) > ) . intvaluef ) ; 

deleteRulesWi thRuleField ( RulelD, FieldID) ; 

! 

10 ; 



//**** * * * *** **** 

15 // Delete All Enhancments with matching RulelD, FieldID 
// Return Vector of afffected Rules 
//** • 



40 



45 



50 



oubiic vector deleteRulesWi thRuleField ( int RulelD, String FieldID: 

20 

Log .instanced . wri teLog { " FlowManager . dele teRui eswi thRuie Field" , 
LogEven: . LOG_DE3UG. "Delete Enhancements, with Field ,, + FieldID) ; 
vector ResultVai « new vector () ; 
25 OrderedSet Af f ectedRules = new OrderedSet ( false } ; 

vector ToDel = new Vector () ; 
Enhancements EnhancementList = CEM . instance ( ) . get Enhancements ( ) ; 
for ( Enumeration E = EnhancementLis t . elements i) ; 
E.nasMoreEl erne nts i ) ; . ) 

30 ,: 

Enhancement Cur ren tEnhancement = ' 
: Enhancement i ; E . next Element ( ) ) ;; 

iz ; ;' FieldID. equals ( CurrentEnnancement . cetFielilDt '; 3 
-; (RulelD -- CurrentEnhancement . cetRulelD ( ; 
35 //( (RulelD == Cur ren t Enhancement . gee R.ulel D ( ) ; II (RulelD 

« -1) ; ) 

{ . • 

ToDel .addElement (Cur re n tEnhancement ) ; 

} ■ 



for tint i=0; KT.oDei . size ( ) ; 

Enhancement CurrentEnhancement = i Enhancement ) Tc Dei . element At ( i ) , 
Af f ectedRules . add < new Integer ( CurrentEn nance me r.t . get Ru i e I D ( 5 



Log . instance t ) .wri teLog ( "FlowManager . deleteRulesWi thRuleField" ', • 
LcgEvent . LCG_DEEUG, "Deleteing "+Cur rent Enhancement ) ; 

try ( Cur rentEnhancement . delete (); } catch (Exception e){} 
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// Set Result Val . 

for ( Enumeration AR * Af fectedRules . elements O ; 
AR . ha sMc re Elements l ) ; ) 

i 

Integer RulelDObj - ( Integer) AR. next Element { ) ; 
ResultVal.addElement (RulelDObj ) ; // add Rule to Result Set 



5 



10 



30 



50 



return P.esultVal 



15 // deleceRuleEnhancements 



public void deleteRuleEnhancements ( int RulelD) 

20 vector ToDel = new Vector'.); 

Enhancements EnhancementList = CEM . instance ( ) . get Enhancements i ) 
for (Enumeration E = EnhancementList . elements () ; 
E . hasMcr eElements ( ) ; ) 

[ 

25 Enhancement Cur ren tEnhancemen t = ( 

'< Enhancement > (E . nextElement O ) 5; 

if ( CurrentEnhancement . getRulelD ( ) RulelD i 



ToDel . addElement (CurrentEnhancement) ; 



for ; : n z i =0 ; i < To De i .size i i ; 



Enhancement CurrentEnhancement = ! Enhancement : Tc De 1 . e i ementAt > 
35 try 

I ■' 
CurrentEnhancement. delete () ; 

} 

catch (Exception e) 

40 i 

Log . instance i ) . wr i teLog i new LocDebug ; "Delete Rule 
Enhancement: Filled... " + el) 

Debug. writeStackTrace i'TlowManager . deieteRuleEnhancements" , e : ; 
45. I 
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/ Delete Ail E r.r. a ncments wich matching ISID 
,'/ If All enhancements per ruie are deleted, disable the Rule 
// If Trigger is Deleted, Delete Rule 
// Return Vector of afffected Rules 

5 //•••• * *** * ** *• 

public vector deleteRuleswi thl S ( int ISID) 

Log. instance (). writeLoc inew LocDebug ( "Deiece Enhancements for "* ISID)) ; 
10 vector Resultval = new Vector ( > ; 

OrderedSet Af f ectecRules = new OrderedSet ( false) ;. 

GrderedSet Af f ectedRuleFields - new OrderedSet ( false ) ; 

Vector ToDel = new Vector ( ) ; 

// Loop on all Rules 
15 Rules RuleList = CEM. instance (). getRules () ; 

for (Enumeration El - RuleList . elements () ; El . hasMor eElements ( ) ; . ) 

( 

// Compare Trigger ID to the ISID being deleted 

Rule CurrentRule = { ( Rule M El . next Element <! ) ',; 
20 me Rule ID = CurrentRule . getRuielD () ; 

if {CurrentRule . getfriggerlD < ) ISID) 

{ 

// yes. found a match. Add to Affected List 
// and delete the whole Flow 
25 Af f ectedRules . add ( new Integer ( RulelD i ;; 

ToDel . addElement (CurrentRule) ; 
deleteRuleEnhancements < RulelD) ; 

) 

I 

30 // Delete from Collection 

:or nr.t i = 0 ; i<ToDel . size I ) ; i + + ) 

Log . ins nance i ) . writeLog < new LocDebug ( "Delete Pule with Tricqer 

35 Rule CurrentRule = ( Rule ) ToDel . elementAt ( i ) ; 

cry { CurrentRule. delete (); ) catch (Exception e){} 



40 



// Loop on All Enhancements, find Rule/Feild pairs affected. by IS 

removal 

Enhancements Enhancement Lis t = CE^ . ins tance ( ) . ge tEnhancement s { ) ; 
for ( Enumeration E2 = EnhancernentLis t . elements i ) ; 
E2 . hasMoreElements ( ) ; ) 

45 Enhancement Cur rent Enhancement = « 

(Enhancement) ; E2 . nex tElement ( ) ) ■ ; 

if (CurrentEnhancement . getlSID ( ) " ISID) 
j 

Log . instance l ). writeLog (new LogDebug ( "Adding Ruie Field to 

50 Affected List " + 
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Current Enhancement . ge^RuielDt )+ " : "- 

CurrentEnhancement . get Fiel'dlD ( ) )); 
5 Af fectedRuleFieids . add (new Pair (new integer 

(CurrentEnhancement . getRuleID{ ) } , 

CurrentEnhancement . get FieldID { ) 5 ); 

■ ) 

10 j 

for (Enumeration E3 = Aff ectedRuleFields . elements.! ; ; 
E3 . hasMoreElements ( ) ; ) 

{ 

15 Pair CurrentPair = ( ( Pair )( E3 . next Element ( )) ); 

// Delete All Enhancemnts for this Rule & FieldID 

deieteRulesWithRule Field ( f ( Integer ) Current Pair . first) . ir.c Value ( )., 
■String) CurrentPair . second ) 
20 AffeccedRuies.add! { Integer) Current Pair. first J; 



// Set Result Val * 
for ( Enumeration AR . - Af fectedRules . element s () ; 
25 AR . hasMoreElements ( J ; ) 

[ 

Integer RulelDObj = ( Integer ) AR . nextElement () ; 

ResultVal .addElement (RulelDObj } ; /-/ add Rule to Result Set 



// Return Af fee tea Rules 
return ResultVal 



35 // • • 

/./ Delete from the given Rule all the Fields that have Enhancements using 
/■/ at least 1 from the given set of Trigger Input Keys . 
■//•♦ • *** # * ' * *■* 

40 • 

public OrderedSet de leteRuleFieldsWi thT ri gger Keys tint F.uield. Vector 
Trigger Keys i 

{ ■ . 

Log. instance M . wr i teLog ( new LogDebug 
45 i "delereRuler ieldsWi thTr iggerKeys " "P.uleld«=" * Ruleld* 

", Tr iggerKeys = " * TriggerKeys 5 ; ; 

int OpNum = 0 ; 

OrderedSet AffectedRule *= new OrderedSet ( false ) ; 
OrderedSet Af f ec tedFields = new OrderedSet ( f alse ) ; 

50 
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if ( TnggerKeys==null !! TriggerKeys . si :e (I -"0 ! 
return null; 

Leg . instance O .writeLog (new LogDebug ("Add Rule to Affected List " + 

Ruleld) j ; 

Af f ectedRule . add (new integer ( Ruleld) ) ; 
//Get all Enhancements 

Enhancements EnhancementLis c = CEM . instance < ) ..get Enhancements ( ) ; 
for (Enumeration E = EnhancementLis t . elements () ; E . ha sMo re Elements ( ) ; ) 



Enhancement CurrentEnhancement = ( ( Enhancement )< E . next Element () ) ); 

if (CurrentEnhancement . getRulel D( ) ==RuleId) 

// Search InputKeys for Trigger ones (OpNum = 0) 

{ 

15 EnhanceStruc.t CurrentEnhanceS truct * new 

EnhanceSt ruct (CurrentEnhancement) ; 

vector ParsedOpKeyln = CurrentEnhahceStruct . get ParsedOpKeyln ( ) ; 

/// • 
int CurrentOpNum; 
20 mt CurrentOpKey; 

for (int i=0; i< ParsedOpKeyln . size ( ) ; i + = 2) 
( 

CurrentOpNum = 
! ( Integer ) Pa rsedOpKey I n . element At ( i )). int Value () ; 
25 CurrentOpKey - 

{( Integer ) ParsedOpKeyln . element At ( i + 1 ) ) . int Value ( ) ; 

if ( CurrentOpNum==OpNum & & TriggerKeys . con ta ins ( new 
Integer (CurrentOpKey) ) ) 

i 

30 Log . instance ( i . writeLog ; new LogDebug ( "Accino Rule Field to 

Affected List 

rurrentEnhancer.ent . cetP.ulelD ( ) *" : " + 

35 CurrentEnhancement . get FieldID ( } )>; 

Af fee ted Fields . add ( Cur rent Enhances t rue t . get Field ID ( ) ) ; 
break; //pass to next Enhancement 

\ 

) // for over ParsedOpKeyln 
40. ) //if (CurrentEnhancement . getRulelD ( )==RuleId) 

|//lccp over Enhancements 

f or ( Enu.T.erat ion S « Af fee ted Fields . element s () ; E . hasMoreEl ement s ; ) ; ) 
i 

45 String Fieldld = ! S t r i ng ) E . next Element ( ) ; 

// Delete All Enhancemnts fcr this Rule £ FieidID 
deleteRulesWi thRuleField < Ruleld, Fieldld; ; 

) 

return Af f eccedRul e ; 
50 J Z/deleteRuleFieidsWi thTr iggerKeys 
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// Delete in all Rules all Fields including an Enhancement from the given 
// ISID with Functionld equal to one from the given set. 



- / « 



public CrderedSet deieteRulesFieldswi thISFunctions tint ISID , Vector 
ChangeaFunctions ) 

Log . instance (). writeLog (new LogDebug 
10 < "deleteRulesFieldsWithlSFunctions " + "ISID'-" + IsrD+ 

", changedFunctions="+ChangedFunctior ; s )); 

vector ToDel= new Vector!) ; 

OrderedSet Af f ectedRules = new OrderedSet ( false) ; 
OrderedSet Af f ect ed Fields = new OrderedSet < false ) ; 

15. 

if ( ChangedFunctions=«null II ChangedFunct ions . si :e ( ) ==0 ) 
return null; 

Enhancements Enhancements t = CEM. instance I ) . get Enhancements ( ! ; 
;cr (Enumeration E = EnhancementLis t . element s I ) ; Z . hasMoreEl eraents ( i ; J . 

20 

' Enhancement Cur rent Enhancemenc = ( < Enhancement )( E . nexiElemen: « J ! } ; 
if (CurrentEnhancement.qetlSIDt) == ISID && 
ChangeeFuncticr.s. contains < new integer {Cur rentEnhancement . getFunctibnID ( ) ) ) ) 

25 / /TbDe 1 . addEl ement (Cur rent Enhancement ) ; 

Log. instance () . writeLog (new LogDebug { "Adding Rule Field to 
Affected List "+ 

Cur rentEnhancement . cetRul elD O- 1 -" " + 

30 

Cur rentEnhancement . cetFieldlDt ) )): 

Af f ectedFielcs . add ( new Pair '.new Ir.teger 

; C- r r er.t Enhancement .. ~e tRu lei D '. ) ) . 

35 Cur rentEnhancement . get Fie ldl D < ) ) ); 

\ 

for(Enumeration E2 - Af fectedFields . elements () ; E2 . hasMc reEl ements ( ) ; ) 
{ 

40 Pair CurrentPair «= ( ( Pai r ) { E2 . next El ement i ) ) ); 

// Delete All Enhancemnts for this Rule FieldID 

deleteRulesWithP.ulerieldl ! ( I nteger iCurrentPair . first) .ir.tValuei ; , 

(Strir.c: CurrentPair. second} ; 
45 . Log . instance ( I .writeLog (new LogDebug i" Add r.uie to Affected 

List "-CurrentPair . first) ) ; 

Aff ectedRules .add < ( Intege r ) Cur rent Pai r . first 5 



50 return Af f ectedRul es ; 
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- /deieteRulesFieidsWi this Functions 



10 



20 



25 



30 



//• ** * ** * 

// In ail Ruies find ail Enhancements which met the fllowing conditions: 
// : 3 1 d = <given 1SZZ)> , FanctionID = <one from the qiven list>. 

.'/ For ail these Enhancements the Functionld is replaced with new one.. 
.'/ The "new" Id is placed immidia tel y after the matching "eld" Id in the 
// 2-nd input parameter. 



public OrderedSet editEnhancements Funct lonlDWithFunctionID < int ISID , 
Vector Aff ectedFunctionlds ) 

throws Exception 

15 i 

Loa . instance (). writeLog (new LogDebug 
{ "editEnhancementsFunctionlDWithFunctionID "+ " ISID="+ISID+ 

", Af f ectedFunctionIds="+Af f ectedFunctionlds )); 
CrderedSet Aff ectedRuleSet = new OrderedSet ( false > ; 



if l Pit f ectedFunction!ds==null li Af f ectedFunctionlds . site l ).==0 I 
{ 

Log . instance ( ) . wr i teLog ( new LogDebug 
{ "edi t£nhancementsFunctionIDWithFunctionID: ", "There are no changed function 

IDs" ) . ; 

return null; 
} 

Enhancements Enhancement Lis t = CEM .instance ( ) . ge tEnhancements ( } ; 
for (Enumeration E = EnhancementList . elements () ; E . hasMoreElement s ( : ; i 

Enhancement CurrentEnhancement = f i Enhancement )( Z . r.ex tElement ( ) ) 
for tint i=0; i<Aff ectedFunctionlds .size <); i+=2 5 



if { Cu r rent Enhancement . getFunctionID ( )= = 
35 ( i Integer J Af fee ted Funct ion Ids . element At ( i ) ; . int Value ( i ) 

( . 

int OldFuncId = Current Enhancement . get Funcr.ior.ID () ; 
int NewFuncId = 
C ( Integer i Af f ectedFunctionlds . eiementAt ( i )). mtValue () ; 
40. Log. instance () . writeLog (new LogDebug ("Replace Functionld 

"+01dFuncId + 

"with " + Newfuncld+ " in Enhancement 

= " +Cur rentEnhancement ) ) ; 

CurrentEnhancement . edit ( ) ;. 
45 CurrentEnhancement . setFunctionID (NewFuncId) ; 

CurrentEnhancement . update ( ) ; 

Log . instance (). writeLog ( new LogDebug < "Add Rule to Affected 
List " -CurrentEnhancement . getRulel D ( } :■ J ; * 
Aff ectedRuleSet . add (new Inteqer 
50 .CurrentEnhancement . cetRulelD ( ) ) ) ; 
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5 return Af f ectedP.uleSec ; 

'» 

} //class 

/ / * • * ****** 

10 // Internal Helper Classes to support flow computations 
// * * 

// * ****** 

15 // 

// * * ******* 

class EnhanceOpSLiSt extends SList 

20 

//*•*• ' * * * * 

/ / 

25 // * •**..***.****. 

public EnhanceOpSLis t ( ) 
super ! j : 

30 



.0 // 



//* 



public EnhanceOpSLis t ( int n) 
40 i 

super ( r.J ; 



45 //• 



50 - 
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30 



35 



40 



45 



50 



p'jbiic Zr.hanceOpSLisr ( EnhanceGpSLis t s ; 
super (si; 



public EnhanceOpNode findOp tint OpValue, String OpField) 
( 

15 for (int i = 0; Ksize ( ) ; i++) 

I 

EnhanceOpNode Op - ( EnhanceOpNode) at < i) ; 

if (Op- getOpNun»( ) == OpValue) 

I 

20 // Field does not matter if we are searching for 

OpNum ; 

•CpValue == 0/ ■; 

( 

25 return Op ; 



if ( (OpField . equals ( Op . ge t FieidI D ( ) :■ » .11 



return null 



public EnhanceOpNode findGuon (int GuonValue) 

i 

:or nr.: i=0 ; i <s x;e ( ) ; i + + ) 

EnhanceOpNode Op = ( EnhanceOpNode ) at (: 
if (Op . getGUON ( ) GuonValue) 

return Op ; 

} 

! 

return null ; 
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//** 

// Object t. K .at represents a sincie Input Parameter of a single. 
1 0 Enhancement 

// Operation. 

// Everv Enhancment should maintain a set of OplnputParam 



15 



20 



25 



class OpinputParam 



{ 



//Original Opnum of Enhancment owning the Input Param 

public int m_OpNum = 0 ; 

.'/ ArgKey Value of the Output Parameter 

public i r.z m_KeylD = 0 •' 

// GUCH value of Enhancment cwnmq the Input Param 
public in: rn_OpGUON = 0 ; 

/ Index of Input Parameter. Defined from the "UMIR Allocation- 
public int m_Key Index - 0 ; 



//////////// 
// Constructors 

////////////. 

public OpInputParamt int OpNum, int KeylD, 



int GUON) 



in_OpNum - OpNum 

m_KeyID = KeylD 

m_OpGUOM - GUON ; 

m_Key Index = 0 ; 



40 



45 



//////////// 

// Get Set 
//////////// 
public int getGUONO 
public void setGUONdnc value) 
public mt getKey<) 
n_Keylp; ) 

public void set Key i int Value ) 

public GuonKeyParam gecGuonKeyi) 
Integer t m_ KeylD; , new integer (m_OpGUON) }; } 



{return m_OpG U OK ; I 
jm_OpGUON = value; i 

{return 

(m_KeyID Value; i 

{return new GucnKey Pa ram l new 



50 



public :r.c get Index < ) 

public void setlndex ( int Value) 



(return m_r'.eylndex; I 
{m_KeyIndex - value; » 
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class GuonKeyParam extends Pair 

public int m_GUON - 0 ; 
15 public int m_Key » 0 ; 

public GuonKeyParam () 

< 

super (.) ; 



public GuonKeyParam (Integer Key, Integer GUOH) 
! 

super (Key, GUON1 ; 



25. ) 

public int hashCodeO 



Integer nl = ( Integer) first; 
Integer n2 = ( Integer) second; 

return t ( n 1 . int Value ( J 16)+ n2 . ir.tvalue ( ) ) 



public boolean equals (Pair pair) 

'35 i 

Integer nl = ( Integer ) f i rs t ; 

Integer n2 = ( Integer ) pair . first ; 

. integer n3 = ( Integer ) second; 

Integer n4 = ( Integer ) pair . second; 

40 return ( c" nl . intValue i ) n2 . intValue u . ss 

p.3.int.'.*aiue(i -n4 . intVaiue i ) > ) ; 



45 



// Object that represences a single Output Parameter cf ; single 
50 Enhancement Operation. 
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// Every Enhancment should maintain a set cf OpOutput Pararr* 



class OpOutput Param 

// Set of Key/GUON pairs, elements are JGL Pairs 

public OrderedSet m^KeyGUONLisr - new OrderedSet ( false ,; // Mo 

Dupl icates 

// Set of Keys, elements are Integers 

public OrderedSet m_KeyList = new OrderedSet ( false j; // Mo 

Duplicates 

// Set of GUONs. elements are integers 

public OrderedSet m_GUONList - new OrderedSet ( false >; // No 

Duplicates 



Get Sec 



public Enumeration getGUONs() 
public Enumeration getKeys ( ) 
25 m_KeyList . elements (); } 

public Enumeration getGUONKeys ( ) 



! return rn_GUONLisc . elements ( ) ; I 
(return 

( return m_KeyGUONLis t . elements { ) ; } 



//public Keys getGUONKeys (GUOH1 . 
//public G'JONs getKeyGUONs < Key; , 



// 

// * ** 

public void addKeydnt Key, int GUON ) 

40 

Pair KeyGucn = new GuonKeyParam (new Integer (Key ),' new 

Integer (GUON) ; > 

m_KeyGUONList . add (KeyGuor. j ; 
m_KeyList . add (new integer (Key) ) ; 
45 m_GUONList .add (new Integer (GUON) ) ; 



S3 



50 



/ / 
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public byte(] getOpKeyOutAsArray ( > 

5 try 

( 

Arguments args = new Arguments ( ) ; 

tor (Enumeration EK = gecKeysO ; EK . hasMoreElements ( ) ; ? 

( 

]0 Integer outKey = ( Integer ) EK . next Element ( ) ; 

args . put ( ArqKeys . KEY_KEY_I G , new 

UNIRShort (( short) outKey . intValue ( ) J); 

) 

15 return args . toByteArray ( ) ; 

} 

catch (Exception e) ( ) 
return null ; 



25 



// Object that representes a single Enhancement Operation. 
// Flew Manager should maintain a set of EnhanceOpNode 

30 //••• * ; ' 

: Uss EnhanceOpNode extencs Enhances t rue: 

*//' Unique Identifier of the Op. Created at Runtime. "Giobai 

Jl? Unique Gp Num" 

int t,_GUON = 0 ; 

// String Description of the Operation, Created "Recursively" at Runtime. 
String m_Descr ipt ion = null ; 

40 // OrderedSet version of Enhances truct . OpKeys In . Each element is 

cf type OpInputParam 

OrdereaSet rr._Tr ans ia cedOpKeys i n = new OrderedSet • false ?; // ::c 

Duplicates 

vector m_TranslatedOpKeysinVector = new Vector (} ; 

45 

// List of Fields that are the "End Product" cf this GGGN . null 
in mcsc objects, has value only e'er 

// "End" Objects. Becuase of Optimizations, one GUON can feed 
more tr.an one field. Elements are Strings 
50 // that were the FieldID of Original Enhancement. 
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ZrrderecSet :r,_Cut Fields - new OrderedSet! false > ; 

// Object that manages the lists of output param keys of this Enhancment 
OpOutputParam Tr._OpKeysOut = new OpOutputParamH ; 

// ISM, Gatherer l Node ID of this (denvea from Enhances true t . I S I D) 

mt m_ISMID = 0; 

int m_GathererID = C 

mt m_NodeID - 0 

// Flag if this GUON was originally the "last" (Last OpNum for a' FIELDID) 
boolean m IsLast = false ; 



10 



/./ The list of EnhanceOpNodes can be viewed as a Directed Acyclic 
15 Graph (DAG), with the GUONS as vertices 

// and the pairs of (GUON n, GUON m using output of n as Input) 

as edges . 

// The Following are Variables Used in a TCPLOGICAL SORT using 
Depth. First Search (DFS ) . 
,20 final static int WHITE - 0 ; 

final static int GRAY - 1; 
finai static int BLACK = 2; 
public int m_ColorStatus = WHITE; 

public EnhanceOpNode m_?i - null; // Predecessor Mode. if. 

25 null, then rc-cc (Several roots possible) 

public int m_DiscoverTime = 0 ; // Start Time of DFS on this. 

GUON 

public int m_FinishTime - 0; " End Time of DFS 

on this GUON :and adiaceny list) 

30 

//////////// 

/ / Constructors 

./////////// 

public EnhanceOpNode ( Enhancement Z) 

35 ( 

super iZ) ; 

} 

public EnhanceOpNode (int RulelD. String FieldID, short Cptium. int ISID. 
40 int Fur.ctionID, byte() OpKeyln, 

Vector ParsedOpKeyln 

{ 

super (RulelD, FieldID. OpNum, ISID, functionID, CpK.eyln. 
45 ? jrsecOpKeyl n ; : 

i 

//////////// 
// Operations 

50 //////////// 
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5 // Create a list cf OpInputParam in m_TranslatedOpKeysIn from the 

pair 

// arguments in m_OpKeyIn. Add the corresponding GUON for every 



10 



15 



45 



OpNum 



•'/ in the OpKeyln. 



public void translateOpKeyln ( EnhanceOpSLi s t OpList) 
( 

try 
I 

Arguments a = new Arguments { getOpKeylnM J; 

for (Enumeration EK = a . element s () ; EK . hasMo r eElements ( ) , 

20 ; 

short OpValue = '■ i UlU RShort } 
EK . nex-Eiemenc [)). get Short Value D ; 

short KeyValue = MUNIRShortJ 
EK . next Element r ;• } . get Short value () ; 
25 OpInputParam tmp « null ; 

EnhanceOpNode CurrentOp = OpLis t . £ indOp (OpValue . 

m_FieidID>; 

tmp «« new OpInputParam (OpValue, KeyValue, 

CurrentOp. getGUON ( 1 i; 
30 m_TranslatedCpKeysIn . add < tmp) ; 

m_Trans latedOpKeys InVect or . acdEierr.enz ( tmp) ; 

i 

i catch ! Exception EAM ) 

35 



// pass thru mjTransla tedOpKeys I n , replacing occurences of 
40 ol dGUC:: with newGUC?: 



public void Swaplnpu tKeyGUOtl (ir.t oidGUOtl, mr newGUOtJ) 

i 

for (Enumeration EK = m_Transia tedOpKeysIn. elements () ; 
SK . ha sKo re Elements { ) ; ? 



OpInputParam InParan = iOpInput Parami EK . r.extElement c ) . 
50 if (inParam. getGUOM ( ) == oidGUON) 
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Ir.Param. setGUON (newGUON) 



10 



// Add Op to Lists of output of current EnhanceOpNode 



15 



20 



public void addOutputParam (int Key, int GUON) 



m_OpKeysOut . addKey (Key, GUON) ; 



•'/ Get Set 



25 



30 



33 



40 



public int getGUONM 

public void setGUON (int Value) 

public int getGa tnerer I D < ) 

rr_Gatherer ID; ) 

public vcid setGathererlD ! int Value) 
public int getNodelDfl 

.i\__Mode I D; } 

public vcid setNodeiD ( int Value) 

public cooiean ge tLas t-Flag ( } 

public void setLastFIag (boolean, value > 

public 3tring get Descr ipt ion ( ) 

public void setDescription ( String value) 

public int getISMID{) 

m_ISMID;) 

public vcid secISMlDlint Value) 

Value ; J 



{ return m_GUON; ) 
(m_GU0N « Value; } 

{ return 

(m_Ga tnerer ID = Value; ! 

i return 

i ~_Noce ID = Va'l ue ; i 
(return rr._:s~ast ; • 

U_lsLasc- = value.;/ 
(return nn_Descr ipt ion ; } 
( m_Descr ipt i on = value.;} 
{ return 

im ISMID - 



45 



'/ Topology Sort 
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10 



3D 



40 



45 



50 



public :r.c ce:Coior i ) -return 
m_Colcr£ca tus ; i 

public void setColorfint Value) I m — Colors tatus = 

Value; 1 

public void setDFSDiscoverTime t int Value) lm_DiscoverTime = 

Value; \ 

public void setDFSFini shTiine ( int Value) { m_Fir.ishTime = 

va iue ; i 

public void set Predecessor ( EnhanceOpNode Value) {m_Pi=Value; | 



public void addOutFields (String FieldID) 
15 (m__Out Fields .'add <FieldID) ; } 

public void addOutFields (OrderedSet FieldList) {m_OutFields 
= m_OutFie!ds . union (FieldList) ; } 

public OrderedSet getOutFields ( ) (return 

rt_Out Fields ; ) 

20 public Enumeration get Inl terator i ; (return 

rr._Trans iaz edOpKeys I nVector . elements (); } 

public Enumeration getOutputGUONs ( ) (return 
m_OpKeysOut . getGUOMs ( J . ; i 

public Enumeration getOu tputGUONKeys ( i {return 
25 m_OpKeysOut . getGUONKeys ( ) ; ) 

public Enumeration getOutputKeys ( ) {return 
m_OpKeysOut . get Keys ( ) ; I 



:' ' Convert OpKeyln structure r.c a oyte array rc 

u::iRIr.struccicn 



public byte() getOpKeylnAsAr ray ( ) 
I 

cry 

Arguments arcs = new Arguments < :■ ; 
for (Enumeration EK = getlr.I terator ( i ; 

ZY.. hasMc reElement s ( ) ; ' 

( 

Oplnput Pararn InParam = 
(OpXnputParam; EK. nextEiement 1 3 ; 

args . put ( ArgKeys . A RG r' E V _E ! * H ANC EME MT _0 ? _ N U 
new UHIF.Shorti ; short ) InParam. m_Key Index ) ) ; 
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args .put I ArgKeys . KEY_KE r , new 

UNIRShcrc ! ( short) InParam.m/.eylD) ) ; 

) 

5 return args . toByteArray () ; 

catch (Exception e)il 
return null ; 
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} 



// * * * * * * ****** 

15 // Convert OpKeyOut structure to a byte array for the 

UNIRInst ruction 

// " **** ******* 

20 public byce{) getOpK.eyOutAsArray i i 

return m_OpKeysOut . ge tOpKeyOutAsAr ray < ) ; 

25 

//*•••*•*•• **** * 

// String representation of this Enhancement 

30 ,</ • — • 

public String :cStrinci ) 

String s = m_GUON + " : " ; 

35 

for (Enumeration EK = ge t Inl tera to r< J ; EK . hcsMorell ement s ( ) ; ) 
( 

OpInputParam inParam = (Op Input Pa ram) EK . r.ext Element ( ) ; 
s += + InParam. m_OpGUON ":" + 

40 InParam . m_OpNum + ":" + InParam . m_Key I ndex + " : " + InParam. n_Key ID+ " ) " ; 



s += "] . {" ; 

45 

for (Enumeration EK2 = m_OutFieics . elements i) ; 
EK2 . hasMoreEl enterics ( i ; : 
i 

Strinc CutFld = < S t ring » EK2 . next Element •. ; 
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s » t " «-outrid+" • " ; 



for (Enumeracion EK3 = getOutpucGUONKeys < > ; 
EK3 . hasMoreElemer.es () ; ) 

Pair GuonKey = ' Pair ) EY.3 . next Element { ) , 
JO s += " ("+GuonKey+") "; 



s += "1 , " + 

m_ISMID + ":" + m_GathererID + ":" * m_NodeID 
return s ; 



20 



50 



// to support special operation of slisc 



public boolean equals ( Obj ect ob j ) 

.; 

return ( m GUON == i ( EnhanceOpNode ) ob j ) . cetGUON ( ) ) - 



35 to support special operation of siis: 



■//*** * * * * * 

public int hashCode() 

40 

//byte vi = (byte) im_GUCtl & 0x000000 ff> ; 
byte vl = 0 ; 
int v2 = 0 ; 

if .'.!n_FieldID. equals »' ■ ! 
45 v2 = m_FieXdID.hashCocet ) i OxOOOOffff 

return ( vl << 24) + (v2 << 3) + n_OpNum ; 

■ i 
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CLAIMS 

What is claimed is: 

1 . A system for tracking network session information, the system 
comprising: 

5 an information source module having a source information input and a 

standardized information output, a source information corresponds to 
network usage information, a standardized information corresponds 
to the network usage information transformed into a standard format; 
a first program having at least a first standardized information input and. 

1 0 an enhanced data output, a first standardized information input 

corresponding to the standardized information, an enhanced data 
corresponding to the standardized data after at least a partial 
transformation, the at least partial transformation being defined 
according to a data record format; 

15 a second program having at least a first enhanced data input and a data 

record output, the first enhanced data corresponding to the enhanced 
data, a data record corresponding to the first enhanced data, the data 
record being formatted according to the data record format; 
a database storing the data record; and 

20 wherein the second program merges duplicate data records that represent 

the same network usage information. 
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2. The system of claim 1 wherein the at least partial transformation 
is defined from a data enhancement procedure, and wherein the data record 
format includes a plurality of fields and wherein the data enhancement 

5 procedure defines how the standardized information is to be transformed into 

the plurality of fields of the data record format. 

3 . The system of claim 2 wherein the data enhancement procedure 
includes at least a field enhancement wherein the field enhancement defines a 
source for a predetermined field in the plurality of fields. 

10 4. The system of claim 2 wherein the data enhancement procedure 

includes at least a field enhancement wherein the field enhancement defines a 
function to be applied to at least a portion of the standardized data. 

5. The system of claim 2 wherein the data enhancement procedure 
defines a plurality of field enhancements, wherein each field enhancement 

1 5 defines network usage information to be stored in the plurality of fields. 

6. The system of claim 5 wherein at least a first field enhancement 
corresponds to an information source module that performs aggregation on at 
least a portion of the network usage information to be stored in at least a first 
field in the plurality of fields. 

20 7. The system of claim 6 wherein the aggregation occurs by 

aggregating packet flow information, the packet flow information corresponding 
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to a plurality of packets, where each packet in the aggregated packets has the 
same IP source address, destination address and port information. 

8. The system of claim 5 wherein at least a first field enhancement 
corresponds to a set of information source modules that performs filtering and 
aggregation on at least a portion of the network usage information to be stored 
in at least a first field in the plurality of fields. 

9. The system of claim 5 wherein at least a first field enhancement 
corresponds to a set of information source modules that performs merging, 
filtering, and aggregation on at least a portion of the network usage information 
to be stored in at least a first field in the plurality of fields. 

10. The system of claim 5 wherein at least a first field enhancement 
corresponds to a set of information source modules that performs event 
notification and provisioning activation. 

1 1 . The system of claim 2 further comprising a second information 
source module, the second information source module having a second source 
information input and a second standardized information output, a second 
source information corresponds to a second network information, a second 
standardized information corresponds to the second network information 
transformed into a standard format, and wherein the data enhancement 
procedure includes a first definition of at least a first field in the plurality of 
fields being from the standardized information, and at least a second definition 
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of a second field in the plurality of fields being from the second standardized 
information. 

12. The system of 1 1 further comprising a first information source 
and a quality of service information source, and wherein the information source 

5 module receives the network usage information from the first information 

source, and wherein the second information source module receives the second 
network information from the quality of service information source, and 
wherein the first definition defines that a source IP address supplied by the first 
information source should be put into the first field, and wherein the second 
1 0 definition defines a supplied by the DNS server should be put into the second 

field. 

13. The system of claim 1 wherein the second program manages the 
first program and the information source module. 

14. The system of claim 1 wherein the second program causes the 
15 data record to be stored in the database. 

15. The system of claim I wherein the information source module is 
configured to receive the network usage information from a predetermined 
network device. 

16. The system of claim 1 wherein the at least partial transformation 
20 includes policy-based data aggregation which defines how network usage data 

should be aggregated. 
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1 7. The system of claim 1 wherein the network usage information 
includes IP session data. 

18. The system of claim 1 wherein the data format includes a 
plurality of fields including a source IP field, a destination IP field, a source 

5 host field, a destination host field, a service type field, a date and time field, a 

duration field, a total number of bytes field, and a counter field. 

1 9. The system of claim 1 further comprising a customer care and 
billing system coupled to the database, the customer care and billing system for 
accessing the database to generate a bill from the data record. 

1 0 20. A network usage accounting system comprising: 

an information source module coupled to receive network information 

from a network device; 
a gatherer coupled to receive the network information source module, 
the gatherer for performing data enhancements on the network 
1 5 information to create a plurality of data records; 

a central database storing the plurality of data records; and 
a central event manager coupled to receive the plurality of data records, 
the central event manager merging duplicate records in the plurality 
of data records, the duplicate records representing the same network 
20 usage information. 

21. The system of claim 20 wherein the information source module 
is configured to receive network information from a network device chosen 
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from the group of network devices consisting of a proxy server, a domain name 
service server, a firewall, a RADIUS server, and a router. 

22. The system of claim 20 wherein the gatherer performs filtering 
and aggregation on the network information. 
5 23. The system of claim 20 wherein the plurality of data records 

have a predefined data format comprising a plurality of fields, and wherein the 
data enhancements includes at least a first data field enhancement to enhance 
the network information to fill in the first data field. 

24. The system of claim 23 wherein the first data field corresponds 
10 to a source IP address field and wherein the data enhancement includes 

extracting a source IP address value from the network information. 

25. The system of claim 23 wherein the first data field corresponds 
to a domain name field and wherein the data enhancement includes requesting a 
domain name from a domain name service server. 

1 5 26. A method of gathering and aggregating network usage 

information from a set of network devices, the system using at least a first 
program and a second program coupled in communications, the method 
comprising: 

accessing network communications usage information; 
20 filtering and aggregating the network communications usage information 

using the first program; 
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completing a plurality of data records from the filtered and aggregated 
network communications usage information, the plurality of data 
records corresponding to network usage by a plurality of users; 
storing the plurality of data records; and 
merging duplicate records in the plurality of data records. 

27. The method of claim 26 wherein completing the plurality of 
records includes accessing user account information. 

28. The method of claim 26 wherein completing the plurality of 
records includes for each data record determining a corresponding source IP 
address, a corresponding domain name, a corresponding type of service used, 
and a corresponding amount of time used. 

29. The method of claim 26 wherein the system includes a third 
program coupled in communications with at least the second program and 
wherein completing the plurality of records includes accessing the third program 
to determine network account information and including the network account 
information in at least a first record in the plurality of records. 

30. The method of claim 26 wherein merging the duplicate records 
includes comparing a plurality of fields in the data records to identify data 
records corresponding to the same network session and merging the 
corresponding records. 

3 1 . The method of claim 26 wherein merging the duplicate records 
includes automatically deleting a duplicate record. 
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32. The meihod of claim 26 further comprising using the second 
program to automatically update the filtering and aggregation performed by the 

first program. 

33. A network usage tracking system comprising: 

5 means for accessing network communications usage information; 

means for filtering and aggregating the network communications usage 

information using the first program; 
means for completing a plurality of data records from the filtered and 
aggregated network communications usage information, the plurality 
I o of data records corresponding to network usage by a plurality of 

users; 

means for storing the plurality of data records; and 

means for merging duplicate records in the plurality of data records. 

34. The network usage tracking system of claim 34 wherein the 
1 5 means for completing the plurality of data records includes one or more 

networked computers running one or more programs. 

35. The network usage tracking system of claim 34 wherein the 
means for storing the plurality of data records includes a relational database. 

36. The network usage tracking system of claim 34 wherein the 
20 means for storing the plurality of data records includes an object database. 

37. A system for accounting for network usage comprising: 
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a plurality of gatherers coupled to receive network.usage information, 
each gatherer for performing data enhancements on the network 
information to create a plurality of data records; and 
a central event manager coupled to receive the plurality of data records, 
5 the central event manager merging duplicate records in the plurality 

of data records, the duplicate records representing the same network 
usage information. 
38. The system of claim 37 further comprising a customer care and 
billing application for receiving the plurality of data records and generating 
10 bills. 
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(57) Abstract 

In some embodiments, network traffic information is captured at network information sources. TTiese sources provide detailed 
MoaM^^^ communications transactions at a network device. Importantly, different types of sources can provide 
Sls o information. Gatherer devices gather the detailed information from the various information source devices and convert 
!^faAM information. T*e gatherer devices can correlate the gathered information ^^ISS^ 
network transaction accounting. Manager devices manage the gatherer devices and store the gathered standardized ,n f°™ a ~ n ; 
SevTces ehminate duplicate network information that may exist in the standardized information. The manager devices also consolida e he 
fnfo^ the information stored by the manager devices represents the consolidated, account correlated, network ~uon 

Information thTt can be used for billing or network accounting. The system thereby provides a distributed network accounting and billing 
system. 
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